JavaScriptApplicationswithNode.js,

React,ReactNativeandMongoDB:

Design,code,test,deployandmanageinAmazonAWS

EricBush

“GivemesixhourstochopdownatreeandIwill

spendthefirstfoursharpeningtheaxe.”

-AbrahamLincoln

BLUESKYPRODUCTIONSINC.

retrievalsystem,ortransmittedinanyformorbyanymeans,withouttheprior

writtenpermissionoftheauthor,exceptinthecaseofbriefquotationsembedded

in evaluation articles or reviews as accompanied by proper references to the

book.

Every effort has been made in the preparation of this book to ensure the

accuracy of the information presented. However, the information contained in

this book is sold without warranty, either expressed or implied. Neither the

book’sauthor,nor publisher,noritsdealersanddistributors willbeheldliable

for any damages caused or alleged, resulting directly by or indirectly by this

book.

Theviewsexpressedhereinarethesoleresponsibilityoftheauthoranddo

notnecessarilyrepresentthoseofanyotherpersonororganization.

Pleasesendanycommentstotheauthoratjsdevstack@outlook.com.

PaperbackISBN-10:0-9971966-6-1

PaperbackISBN-13:978-0-9971966-6-5

e-bookISBN-10:0-9971966-7-X

e-bookISBN-13:978-0-9971966-7-2

Contents
Preface
AbouttheAuthor
Introduction
PARTI:TheDataLayer(MongoDB)
Chapter1:Fundamentals
1DefinitionoftheDataLayer
2Datalayerdesignprocess
3IntroducingMongoDB
4TheMongoDBCollection
5TheMongoDBDocument
Chapter2:DataModeling
1ReferencingorEmbeddingData
2WhentouseReferencing
3ReferenceRelationshipPatterns
4AHybridApproach
5DifferentiatingDocumentTypes
6RunningOutofSpaceinaDatabase
7AccessControl
Chapter3:QueryingforDocuments
1QueryCriteria
2Projectioncriteria
3QueryingPolymorphicDocumentsinaSingleCollection
Chapter4:UpdatingDocuments
1Updateoperators
2ArrayUpdateOperators
3Transactions
Chapter5:ManagingAvailabilityandPerformance
1Indexing
2AvailabilitythroughReplication
3Sharding
Chapter6:NewsWatcherAppDevelopment
1CreatetheDatabaseandCollection
2DataModelDocumentDesign
3TryingOutSomeQueries
4IndexingPolicy
5MovingOn
Chapter7:DevOpsforMongoDB
1MonitoringthroughtheAtlasManagementPortal
2TheBlameGame
3BackupandRecovery
PARTII:TheServiceLayer(Node.js)
Chapter8:Fundamentals
1DefinitionoftheServiceLayer
2IntroducingNode.js
3BasicConceptsofProgrammingNode
4Node.jsModuleDesign
5UsefulNodeModules
Chapter9:Express
1TheExpressBasics

2ExpressRequestRouting
3ExpressMiddleware
4ExpressRequestObject
5ExpressResponseObject
6TemplateResponseSending
Chapter10:TheMongoDBModule
1BasicCRUDOperations
2AggregationFunctionality
3WhatAboutanODM/ORM?
4ConcurrencyProblems
Chapter11:AdvancedNodeConcepts
1HowtoScheduleCodetoRun
2HowtoBeRESTful
3HowtoSecureAccess
4HowtoMitigateAttacks
5UnderstandingNodeInternals
6HowtoScaleNode
Chapter12:NewsWatcherAppDevelopment
1InstalltheNecessaryTools
2CreateanExpressApplication
3DeployingtoAWSElasticBeanstalk
4BasicProjectStructure
5WhereitAllStarts(server.js)
6AMongoDBDocumenttoHoldNewsStories
7ACentralPlaceforConfiguration(.env)
8HTTP/RestWebServiceAPI
9SessionResourceRouting(routes/session.js)
10AuthorizationTokenModule(routes/authHelper.js)
11UserResourceRouting(routes/users.js)
12HomeNewsRouting(routes/homeNews.js)
13SharedNewsRouting(routes/sharedNews.js)
14ForkedNodeProcess(app_FORK.js)
15SecuringwithHTTPS
16Deployment
Chapter13:TestingtheNewsWatcherRestAPI
1DebuggingDuringTesting
2ToolstoMakeanHTTP/RestCall
3AFunctionalTestSuitwithMocha
4PerformanceandLoadTesting
5RunningLint
Chapter14:DevOpsServiceLayerTips
1ConsoleLogging
2CPUProfiling
3MemoryLeakDetection
4CI/CD
5MonitoringandAlerting
PARTIII:ThePresentationLayer(React/HTML)
Chapter15:Fundamentals
1DefinitionofthePresentationLayer
2IntroducingReact
3ReactwithonlyanHTMLfile
4InstallationandAppCreation

5TheBasicsofReactrenderingwithComponents
6CustomComponentsandProps
7ComponentsandState
8EventHandlers
9ComponentContainment
10HTMLForms
11LifecycleofaComponent
12TypecheckingyourProps
13GettingareferencetoaDOMelement
Chapter16:FurtherTopics
1UsingReactRouter
2UsingBootstrapwithReact
3MakingHTTP/RestRequests
4StatemanagementwithRedux
Chapter17:NewsWatcherAppDevelopmentwithReact
1WhereitAllStarts(src/index.js)
2Thehubofeverything(src/App.js)
3ReduxReducers(src/reducers/index.js)
4TheLoginPage(src/views/loginview.js)
5DisplayingtheNews(src/views/newsview.jsandsrc/views/homenewsview.js)
6SharedNewsPage(src/views/sharednewsview.js)
7ProfilePage(src/views/profileview.js)
8NotFoundPage(src/views/notfound.js)
Chapter18:UITestingofNewsWatcher
1UITestingwithSelenium
2UITestingwithEnzyme
3DebuggingUICodeIssues
Chapter19:Server-SideRendering
1NewsWatcherandSSR
Chapter20:NativeMobileApplicationdevelopmentwithReactNative
1ReactNativestarterapplication
2Components
3Stylingyourapplication
4Layoutwithflexbox
5Screennavigation
6Devicecapabilityaccess
7CodechangestoNewsWatcher
8ApplicationStoreDeployment
CONCLUSION
INDEX

Acknowledgements

Thisbookcoverssomephenomenaltechnologyframeworks.Oneofthoseis

Node.js. I am in awe of Ryan Dahl for his having conceived of it in the first

placeandforhishavingbuiltitontopofsomegreatworkbyothers.Thankyou

Ryan!Iwanttothankallthosewhohavecollaboratedonthemodulesthatmake

Nodesuchacapableplatform.Thisisopen-sourceatitsbest.

I would also like to thank TJ Holowaychuk for his contribution to node

through his work in creating the Express web framework. This really does a

greatjoborganizingandsimplifyingthecodeneededtoimplementaRestWeb

API.

IwanttothankthebrilliantpeoplebehindalloftheAmazonWebServices

infrastructure.Fromthosewhoenvisioneditinthefirstplace,tothosewhobuild

itandalsothosewhorunit24-hoursaday.Everyyearitaddsamazingplatform

services for all to take advantage of. This liberates application developers to

code up their applications to provide customer value and benefit from many

underlyingservicesthatdoalotoftheheavylifting.

AnothergreatpieceoftechnologyisthatofMongoDB.Iwanttothankthose

who develop and maintain MongoDB and also thank the Atlas team for their

makingiteasytohostMongoDBinaPaaSenvironment.

I acknowledge all the effort that is ongoing to build, maintain and evolve

ReactandReactNative.TheReactframeworkfromFacebookisaperfectfitin

theJavaScriptstack.

IexpressmylovetomysweetheartLoradel.Thankyouforfindingme.Iwill

cherishyouforever.

Preface

Manypeoplewillturntothefirstfewpagesofabooktodeterminewhether

or not it will meet their needs. If you are just now making a decision about

whetheryouwanttoinvestthetimeandmoneynecessarytobuildupyourskills

intheareaoffull-stackdevelopment,Iwillstatethatyouaredoingwelltohave

foundthisbook.Letmegiveyouabriefmarketingpitchaboutwhyyouneedto

pursueinvestinginfull-stackJavaScriptskills.

Technologytrendscomeandgo,butwhatthisbookcoversisreallyhereto

stay.TheJavaScripttechnologiesthatthisbookcovershavecometogetherina

unified stack, all using the JavaScript programming language. With this book,

you will not only learn about these technologies, but will also gain a longer

lastingfoundationalunderstandingofwhatathree-tierarchitecturaldesignis.

IfyouareevenremotelyinterestedinJavaScriptdevelopment,Iadviseyou

todedicatetimetolearnthesetechnologiesanditwillpayoff.Youwillhaveto

study hard, but it will be worth it. There is a huge development community

behindthisandyouwillbeingoodcompany.Manytopcompaniesarefinding

successdoingfull-stackJavaScriptdevelopmenttoday.

Levelofskillrequired

YouwillneedtohavesomebasicunderstandingofJavaScripttobeeffective

inlearningwhatthisbookcontains.Ifyoudon’talreadyknowthebasicconcepts

oftheJavaScriptlanguage,youcanuseonlinecontenttolearnthemoryoucan

buyabooktolearnthem.

Ihavewrittenthisbookforalllevelsofexperience.Ifyouareabeginnerand

needtolearnsomeofthebasicconceptsofathree-tierarchitecture,Ihaveyou

covered.Ialwaysleadoffwithsomematerialthatcoversthebasicconceptsthat

areinvolvedinthearchitecture.Idothissothatwhenyougettotheactualcode

implementation,youwillhavethisinsightandarenotjusthaphazardlywriting

code.

Thisbookpullstogetheralotofinformationthatyouwouldhaveahardtime

finding.Oneofmygoalswastogiveyouknowledgethatwillhelpyougetand

keepajob.Ihaveputmyselfinyourshoesandhavegivenyoutheinformation

thatyoureallyneedtogaintraction.

IfyouarealreadyfamiliarwiththebasicsoftheMERNstack,thisbookcan

getyoutothenextstageandactuallyleadyoutowardsprofessionalenterprise-

level development. I make sure to dive deep into each topic and show real

implementationanddeploymentdetails.Inotherwords,thisbookisnotacopy

oftheAPIdocumentationofeachframework.

For the experts in MERN development out there, this book may save you

timetryingtofigureoutwaystotestandsecureyourapplication.

Halfthe battleof learninganynew technologyislearning theadd-onsand

toolstogetitallworking.Thisiswherethisbookwillreallycomeinhandy.I

don’tjustdumpalotofcodeonyou,Itakethetimetoexplainthedevelopment

processandwhattoolsyoucanuse.

AWSandotherknowledge

This book utilizes AWS for its implementation. You get to learn about

configuring and deploying to that environment. As mentioned, this book also

contains substantial foundational instruction, such as details on security

measuresandinformationonhowtoperformtesting.

Choices

Thisiscertainlyagreattimetobeasoftwaredeveloper.Oneofthethings

thatmakesitsogreatisthemyriadofopen-sourceprojectsavailabletoleverage.

Thereareframeworksthatexisttohelpyouwitheverylayerofyourarchitecture

fromtheback-endservicestothefront-endGUI.Thereareliterallyhundredsof

options for technologies touse in your development of a full-stack JavaScript

application.

I will help you get started by narrowing things down to some of the

essentialsthatareneededtogetupandrunning.Iwillalsocovermoreadvanced

topicstohelpyoubuildqualityenterpriseapplications.

Ihadtomakesometoughdecisionsonwhichframeworksandtoolstoutilize

inthisbookandhavedonesobasedonthefollowingcriteria:

1. Itmustbesimpleandquicktogetvaluefrom.

2. Itmusthave wideadoption,with committedsupportfor

growthandfuturerelevance.

3. Itmusthavebroadplatformreach.

After mastering what is covered in this book, you can turn around and

evangelize full-stack JavaScript development as a reality today. It is great

alreadyandwillonlygetbetter.

Happycoding,

EricBush

NoteaboutapreviousbookIauthored:Basedontheaboveandmanyother

factors,ImadetheswitchfromAngulartoReactforthisbookandchangedthe

title.Manyotherchangesandadditionswerealsomade.

AbouttheAuthor

EricBushbeganhisprofessionalcareerprogramminginassemblylanguage

for embedded hardware microcontrollers for airplane flight systems. He then

worked developing CAD software on what were known as workstation

computers.WhenWindowscamealong,Ericstarteddevelopingsomeofthefirst

desktop applications for Windows. After that, he transitioned back to client-

serverapplicationsforWindowsServer.Finally,hemadethemovetoenterprise

cloud development in 2009 and is now focused on full-stack JavaScript

technologies,usingMongoDB,Node.jsandReact.

He has worked for some well-known companies including Walmart Labs,

Microsoft,Tektronix,MentorGraphics,Nike,Intel,andBoeing.

Ericworkedasafull-stackdeveloperfortheworldwideretailgiantWalmart.

HeworkedonbackendscalableJavaScriptNode.jscloudinfrastructureandalso

on front-end React code for the walmart.com website store details pages. The

backendworkservesuphigh-trafficproductsearchesandcartcapabilitiesused

by millions of people each day. The work included many of the HTTP Rest

endpoints in use. He was also involved in the DevOps work for Walmart

enterprise practices of scalability, quality, security, reliability, availability and

performance.

As a consulted at DocuSign in downtown Seattlehe was in the role of an

architect/developer and developed an internal enterprise application suite of

toolstoimprovetheefficiencyofemployees.Thisinvolvedfull-stackJavaScript

withHTML/AngularthattalkedtoabackendNode.jsserverthatintegratedwith

aMongoDBdatabase.

AsaconsultantthroughCrosslakeTechnologies(crosslaketech.com)heputs

his considerable breadth and depth of knowledge into helping companies

implement modern PaaS cloud architectures. Crosslake helps companies

envision, plan, architect, develop and execute engineering practices and

architectures to satisfy business and customer needs. Crosslake works to

transform and optimize software delivery. Crosslake also provides services for

technicalduediligenceformergers,acquisitionsandinvestments.

YoucanemailEricatjsdevstack@outlook.com.

Introduction

Thisbookpresentsthetechnologiesandbestpracticesthatspanacrossallof

thelayersofasoftwaredevelopmentarchitecture.Thecoreofthisbookfocuses

on specific JavaScript frameworks used to implement each architectural layer.

Everything is based on MongoDB, Express, React, React Native and Node. I

will walk you through the design and development of a sample application to

teachyouthebestpracticesinarchitecting,coding,testing,securing,deploying

andmanagingaRESTfulWebServiceandSPA(SinglePageApplication).

Thisintroductionsectionwilldefineseveralkeyterms,andintroduceyouto

thesampleapplication.Thisisimportantcontentthatyouwillneedbeforeyou

divedeeperintothematerialspresentedintherestofthebook.

Whatisadevelopmentstack?

Adevelopmentstackisthecollectionoflanguagesandtechnologiesusedto

constructthesoftwareapplication.Thesearethetechnologiesyouwouldpiece

togetherfrombottomtotop.Thisisdifferentfromwhatasoftwarearchitecture

is.Anarchitectureisapatternofcomponentsandlayersforbuildingsoftware.

This book utilizes a particularly popular set of technologies referred to as the

MERN (MongoDB, Express.JS, React and Node.js) development stack. The

architecturepatternfollowedisreferredtoasaSOAthree-tierarchitecture.

MongoDB, Express.JS, React and Node.js are called platforms, or

frameworks,becausetheytakecode,orsomeformofmarkup/configurationand

executeitatahigherlevelofabstractionabovetheoperatingsystemlevel.

TheMERNstackonlyspecifiesfourtechnologies.However,inreality,there

aremanymoretechnologiesthatcomeintoplay.Node.jsasaframeworkreally

gets its capabilities from modular plug-ins that extend its basic capabilities.

Platforms are usually extensible, meaning they are built to be extended in

functionalityby others inthe community thatprovide modules. Youwilllearn

about many of the important extensions that can be utilized to extend the

capabilitiesofNode.js,ExpressandReact.

Note: AWS does have database services such as DynamoDB, a NoSQL

databasetechnology.I havechosento develop withMongoDB instead. Thisis

becauseofitsrichsetofcapabilitiesanditspopularity.MongoDBisstillhosted

inAWSasaPaaSservice,eventhoughitisnotprovideddirectlybyAWS.

Thethree-tierarchitecture

No matter what you are constructing, it is a good idea to modularize and

buildwithlayerssothatyoucanmoreeasilyassembleeverything.Ifyouwere

buildingahouse,youwouldnotbuilditasonejumbled-upmess.Instead,you

wouldfirstlayafoundation,andeventually,constructtheatticandroof.Eachof

the needed modules would fit together: flooring, walls, heating, plumbing,

electrical,etc.Softwareconstructioncanbethoughtofinasimilarway.

Architecting software in layers along with writing code as modules (also

referred to as components) helps with testing, debugging, and incorporating

futureenhancements.Thelatestbuzzwordintheareaofsoftwareconstructionis

“microservices.” You may also have heard of the older concept of a service-

oriented architecture (SOA). These are patterns that encourage layers of

separation.

Oneprovenapproachtodevelopingasoftwareapplicationistodivideitup

intothreedistinctlayers.Thethreelayers(ortiers)arenameddata,service,and

presentation.Youcanalsoutilizepatternssuchasmodelviewcontroller(MVC)

withinathree-tierarchitecture.

Definitionsofthethreetiersarelistedbelow.Anassessmentisprovidedfor

whereeachMERNtechnologyfitsintothoselayers.Whilethislistgivesyouthe

briefestofintroductionstoeachofthese,thethreepartsofthisbookgointothe

detailsofeachlayer.

DataLayer(MongoDB):

This layer is where you persist your data in a DBMS (Database

Management System). Data can be stored, retrieved, updated, or deleted.

MongoDBiscategorizedas a“document-based”database.Thisbook will

discuss a JavaScript API used to interact with MongoDB. Amanagement

portalcalledtheAtlasportalandatoolnamedCompasswillbeused.

ServiceLayer(Node.js+Express.js):

This layer is where a web service API is exposed to contain your

businesslogicandisthedoorwaytothedatastoredinthedatalayer.The

service layer performs workflows that require more complicated

computations and sequencing of operations. These might be HTTP

endpoints or code that is scheduled to run periodically. Node.js was

specifically designed for writing scalable server-side web applications.

Express.jsis used as a Node add-onmodule to simplify the buildingof a

web service layer. Express has support for HTTP request routing and

sendingbackofresponses.

PresentationLayer(React/HTMLandReactNative):

Thislayerprovidesthecapabilitytointeractwithanddisplaydata.This

could be on a computer or mobile device of some type. React provides a

waytocodeupcomponentsthatrenderaUIbyformulatingtheUIcontrols

thatdrivethedisplay.

Note:Inmyopinion,themiddleservicelayerchoiceofNode.jsissomething

foundationaltoaJavaScript-baseddevelopmentstackandcouldnotbeswapped

out. The other choices of MongoDB and React could be swapped out and

replaced with some other similar technology choices. For example, since

MongoDB is accessed through a Node module, you could easily find another

solutionusingwhateverdatabaseyouprefer.Reactisthehighestlayerandcould

bereplacedwithanynumberofframeworksthatallowyoutomakeHTTPweb

requeststotheNodeservicelayeranddothedatabindingwitheitherclientor

server-siderendering.

UsingJavaScripteverywhere

Youunderstandfromthe booktitle,that100%ofthe developmentdonein

thisbookisinJavaScript.Thereareconsiderablebenefitswithhavingonesingle

language used in all three architecture layers. For one thing, the code will all

lookthesameandbesimilarlyrefactoredandmaintained.Youcanalsobenefit

fromusingthesametestingframeworkacrosslayers.Someofthelibrariesyou

downloadcanevenbeusedinbothReactandNode.jscode.

JavaScript has huge momentum with lots of online resources and tons of

available open-source code. You will soon become good at searching online

before you start coding anything. Always search first, just in case there is a

moduleouttherethatwilldowhatyouneed.

Note:YoucandecidetouseTypeScriptwithyourfull-stackcoding.Thecode

fortheusageofNode.jscanstillbethesameasfoundinthisbook,aswellasthe

usageofReactandMongoDB.Theremaybeminordifferences,butthosecanbe

researchedontheinternet.

Note: JavaScript Object Notation (JSON) will be used as the data-

interchangeformat.ThisisanexcellentfitwithaJavaScriptapplication.

Usingpubliccloudinfrastructure

Every piece of code and all data storage should be hosted on public cloud

infrastructure.Thisgivesyoucharacteristicslikefastdeployment,lowcost,and

elastic scalability. This book shows you how to construct an application that

hostseverythinginAWS.

Asfarascloudhostingstrategies,theycanbroadlybebroadlysplitupinto

three categories. These are IaaS, PaaS, and SaaS. The “aaS” part of each

acronymstandsfor“asaService”.

InIaaS,the“I”isfor“Infrastructure”andmeansyoucanutilizethelowest

levelvirtualmachines (VMsor containers)and havecompletecontrol ofwhat

operatingsystemsandsoftwareyoudeploy.

InPaaS,the“P”isfor“Platform”.Thisvariationmakesyourlifeabiteasier

as a developer. It still gives you the flexibility you need over machine

deployment and scaling, but it frees you from the day-to-day maintenance of

machines. You could go through the added work of using IaaS to install and

manage your own Node.js service, or use a PaaS service instead that is much

simpler to setup and operate. After all, you want to spend your time on

applicationdevelopment,notoninfrastructureprovisioningandmaintenance.

InSaaS,the“S”isfor“Software”andindicatesacompleteofferingsuchas

Salesforce.com,AzureSharePoint,orAmazonWorkMail.Thinkofthisasbeing

applications that formerly would have been individually installed on your

machineatworkorhome,butthatnowcanberunningonamachineinthecloud

andaccessedfromanywhere.

Note:Sincecustomersusingacloudservicedon’tknowwhenchangeswill

happen to the service, you want to make sure they have uninterrupted access.

For example, with Elastic Beanstalk as your hosting PaaS, your deployment

updatescanbe“rolling”sothatyourserviceisnevertakendown.Youwilllearn

more about this. AWS also handles OS and hardware upgrades for you in a

rollingmannerthatkeepsyourserviceupwhilethathappens.

MicrosoftVisualStudioCode

Ifyouareoutintherealworldtryingtomakealivingbywritingsoftware,

therealityisthatyouhavetobeabletocombinemanytechnologiesatonceand

targetmanydifferentplatforms.Centraltothisisthecodeeditorthatyouchoose

touse.VisualStudioCode(VSCode)isagreattoolforeditingyourcode.This

isthe tool I used to code and debug the sample applicationused in this book.

Youcancertainlychoosewhatevercodeeditoryoulikeandarenotrequiredto

useVSCode.

VSCodeoffersaricheditingenvironmentaswellasintegratedfeaturesfor

sourcecodecontrol anddebugging. VSCode alsohasthe capabilityto launch

tasksusingtoolslikeGulp,withoutneedingtojumptoacommandlineprompt.

These tools can be used to automate the build and testing steps that you run

frequently.

UsingVSCode,youcancreateaproject,runitlocallyonyourmachine,and

have access to IntelliSense, debugging, git commands and app publishing. VS

CodecanbeinstalledonMacOSX,andLinux,aswellasonWindows.

If you are starting from scratch and don’t have any development tools

installed,goaheadandinstallthefollowing:

✓VisualStudioCode.DidImentionhowcoolthisis?

✓Git.Thisisusedforsourcecodecontrol,butitcanalsobethemeansof

deployingtoAWSinfrastructurethroughGitHubandothertools.

✓ Node.js from https://nodejs.org/. Get a stable version that is labeled

LTS. This will also install NPM (a package manager for JavaScript

projects)atthesametime.

TheNewsWatchersampleapplication

ThecodingconceptsinthisbookareillustratedusingasampleapplicationI

called NewsWatcher. The sample application will help you understand how

everythingcomestogetheracrossallthreelayersoftheapplicationdevelopment

stack. I will now walk you through the capabilities and architecture of the

sampleapplicationcalledNewsWatcher.

The basic capability of NewsWatcher allows a user to set up keywords to

filterthroughnewsstories.Theusercanview,shareandcommentonthenews

stories.Itwillinitiallybedevelopedasawebsitetorunonadesktopormobile

device browser and then the UI will also be written to work as a native

applicationforiOSandAndroid.

It is important to spend a short amount of time on the vision for what

NewsWatcher was meant to be. I needed a vision statement and also some

sketchesofwhatitshouldlooklike.Aroughplanforthedevelopmentwaslaid

out from there, including a prioritized backlog of features. I happen to prefer

followingtheKanbanprocesstoworkthroughtheworkinabacklog.Hereisthe

visionstatementthatIputtogetherforNewsWatcher:

VisionandValuepropositionStatement

Users of NewsWatcher will get news personally served up to them.

NewsWatcherwillbetheirtrustedadvisortoalertthemandsaveimportant

information,withouttheirneedingtoconstantlyscanendlessfeedsofnews

stories.Userscandothingslikeadjustfilterstogetjustthenewstheyare

interested in, see what their friends are looking at, and share comments

aboutnewsstories.

Thisvisionstatementisthelonger-termgoalofwhattoshootforandwould

takemanyiterationstorealize.Fromthisvisionstatement,Iwasabletodistill

thevisiondowninto alistof featuresthatcan beiteratedover.HereiswhatI

cameupwithforNewsWatcher.ThelistcontainssomerequirementsandImay

havesnuckinabitofimplementationdetailalso.

Prioritizedfeaturelistthatfulfillsthevision:

1. Icansetupmultiplenewsfilterstoholdnewsstories.

2. Icangivemyfilteratitletoidentifyit.

3. I can set up keywords for each filter with Boolean operations of

ANDandOR.Forexample,“TreatmentsANDinsomnia”,“DogsOR

Cats”.

4. ThefiltersareperiodicallyscannedbytheNewsWatcherbackend

service to search for news stories that match from a central pool of

collectedstoriesfromanewsserviceprovider.Client-sideprocessing

will be freed up. Finding of stories proceeds even while devices are

turnedofforunabletoconnecttotheinternet.Theserver-siderunsa

periodicpullofstoriesfromanewsfeedservice.

5. Newsstoriesinfolderscanbescrolledthroughandclickedonto

takemetothestorycontent.The20mostrecentstoriesareshownfor

eachfilter.

6. UponopeningNewsWatcher,Iseethehomepageofnewsstories.

7. Icanclickonandopenanewsfoldertoseethestoriesinit.

8. Storiesinfilterfolderscanbesavedtoanarchivefolder.

9. IcandeleteindividualsavedstoriesIselect.

10. Icandeletethefilteritselfwithallofitsstories.

11. My account settings are stored server-side, but cached on the

deviceclient-sidealso.

12. IcandeletemyNewsWatcheraccount.

13. TherearegloballysharednewsstoriesinwhichIcanshareastory

tobeseenbyallNewsWatcherusers.AllotherNewsWatcheruserscan

see and comment on these stories. Stories will be kept for a week

before being deleted. A maximum of thirty shared stories can exist;

newoneswillbumpolderonesoffbeforetheweekisup.Anygiven

user is limited to sharing five stories a week. There can be thirty

commentsperstorykept.Offensivelanguageincommentsisblanked

out.

14. IcanviewmynewsonwhateverdeviceIamloggedinwith.

15. Theloginexpiresandrequiresmetologinagainperiodically.

16. Two-factor authentication requires a text message code to be

entered.

17. I can set actions on news folders to alert or email me when new

entriesarefound.

18. Icansetlimitsonwhennewsalertsaresenttome.Forexample,

waitatleasttenminutesbetweenalerts.

19. IcansettimesduringthedaywhenIdon’twantnewsalertssentto

me.

20. I can set NewsWatcher to download stories to my device when

connectedtoWi-Fi.

21. Icanuseasearchentryboxtodoasearchofallloadedstories.

22. IcandesignatetheorderIwanttoseethefiltersin.

23. I can email stories to others or share them via Twitter and

Facebook.

24. iOS and Android phones have a Native app available in the app

stores.

This book will only implement the core functionality of NewsWatcher in

ordertoproduceaMinimalViableProduct.Noteverythingwillmakeitintothe

initialiteration,butitisnicetohaveabacklogreadytodrawfrom.

Wireframeprototype

The following wireframe images were created in PowerPoint. PowerPoint

hasagreatsetofstoryboardtemplatesyoucanusetopiecetogetheraUIimage

and make realistic looking prototypes. There are also many nice websites you

canusetosketchoutaUIwith.

TocreateaPowerPointprototypeimage:

1. OpenPowerPointandcreateablankslide.

2. Click the Storyboarding tab, then click Storyboard Shapes on

theribbontoopentheStoryboardShapeslibrary.

3. Start creating your prototypes by dragging shapes from the

StoryboardShapeslibrarytoyourslide.

Figure1-NewsWatcherwireframes

Note:IfyouareusingPowerPoint2013,youneedto haveinstalledVisual

Studio 2013 or later, or the Team Foundation Server Standalone Office

Integration 2015 on the same machine to create and modify storyboards.

PowerPoint2016isinstalledwitheverythingyouneedforthisfeaturetowork.

Apeekattheresult

HereareafewscreenshotsofwhattheactualWebUIendeduplookinglike

onamobiledevice.Itdoesn’tlookquitelikethewireframes,asanygooddesign

willevolve.

Figure2-NewsWatcheractualresultingUIonamobilephone

You can try the completed sample browser app on your mobile device or

desktopby going tohttps://www.newswatcher2rweb.com. Download the React

NativeappfromGooglePlay.

AllofthecodeforthesampleapplicationcanbefoundonGitHub.Youcan

download a ZIP file from https://github.com/eljamaki01/NewsWatcher2RWeb.

The React Native code is found at

https://github.com/eljamaki01/newswatcher2RN.

Apeekatthearchitectureanddeploymenttopology

The three-tier architecture of NewsWatcher is depicted in the following

diagram.Thearrowsshowwhatpartsofthearchitecturecallwhatotherparts.

Figure3-NewsWatcherarchitecturediagram

The machine topology required to implement a three-tier application can

vary widely and may even change over time. This is because the topology is

reallyafunctionofthescalingthatyourapplicationneedstoachieve.Youwould

certainly not start off with a topology that could scale to millions of users. It

wouldbeunnecessarilycomplicatedtoprovisionandmaintain.Youdoonlypay

forwhat youneed, however,some things wouldbe inplace thatwill stillcost

youmoneythatyouwouldotherwisenotneed.

Thefollowingdiagramgivesyoualookatastartingpointforanarchitecture

topology. This topology would give you a fair amount of scaling to handle a

largenumberofusers.

Figure4-NewsWatcherservicehostingtopology

PARTI:TheDataLayer(MongoDB)

Thefirstpartofthisbookwillnowinstructyouonwhatadatalayerisand

howtoimplementoneusingMongoDB.ThiswillbehostedusingDBaaS/PaaS

inAWSusingtheMongoDBInc.Atlasoffering.

Theveryfirststepincreatingadatalayerinvolvesthemodelingofthetypes

ofdatayouwillwanttostore.Afterthat,youwilllearnwhatMongoDBisand

howitcanbeusedtoimplementadatamodel.

To finish out the first part of the book, you will actually construct a data

layer that will support the needs of the NewsWatcher sample application. You

will end up with a fully capable data layer that can be utilized by the service

layeroftheapplicationarchitecturebeingbuiltinthesecondpartofthisbook.

Inordertogivefullcoveragetothetopic,Iwillalsocoverwhatitmeansto

managetheday-to-dayoperationsofaMongoDBdeployment.

Chapter1:Fundamentals

Thischapterpresentstheconceptsofabackenddatastoragesystem.Iwill

show you what one is composed of and what capabilities are essential. After

lookingatbackendstoragesystemsingeneral,Iwilldelveintothespecificsof

MongoDBandshowhowitisintegratedintotheNewsWatcherarchitectureto

fulfillthebackenddatastoragerequirements.

Note: This book is about full-stack development. While you will be shown

how to get MongoDB set up in a PaaS environment and learn some basic

development and monitoring, topics like what it takes to fully administer the

serviceare out of scope forthis book. Please seethe MongoDB and theAtlas

helpresourcesformoreinformationonadministrativetopics.

1.1DefinitionoftheDataLayer

The data layer of a software application architecture provides for the

persistent storage of information. Anything of importance can be stored there,

such as customer account information, inventory, orders, audit logs, tax

computation tables, and anything else that can conceivably be stored in

electronicform.

The term database management system (DBMS) is used to describe the

commercial offerings that implement a data layer at the lowest level. Each

availableDBMShascapabilitiesthatdifferentiateitfromtheothers.MongoDB

fromMongoDB,Inc.isonesuch DBMSsystemthatisdevelopedas anopen-

source project as well as being a commercial offering. See

https://www.mongodb.com/ for their various offerings and capabilities.

MongoDBAtlas(https://www.mongodb.com/cloud/atlas)isacommercialPaaS

offeringfromMongoDBInc.Youwillalsofindothercompanies,suchasmLab

(https://mlab.com/)thatalsohostversionsofMongoDBinaPaaSenvironment.

DBMScapabilities

DBMSimplementationscan be placedinto severalgeneral categories.One

possiblewaytocategorizethemisasfollows:relational,key-value,hierarchical,

object-oriented,ordocument-based.SomeDBMSscanalsobesaidtobeofthe

“NoSQL”type.Eachcategoryexistsforaspecificreasonandyouwouldwantto

look at your specific needs and choose the particular DBMS technology that

fulfillsyourneedsthebest.

A DBMS will provide the physical storage medium where the data resides

and should even withstand a power failure. In a cloud-hosted DBMS, data

eventuallymakesitswaytobeingstoredonnon-volatilecloudstoragedrivesin

asecuredatacenter.Datastoragemightevenbegeo-replicatedbetweendistant

datacentersforredundancyandload-balancing.

A DBMS will provide for the storage and access features for the creation,

retrieval, updating, and deletion of data. The acronym “CRUD” is commonly

usedtorefertotheseoperations.ADBMSwilltypicallyprovidethefollowing

features:

Datastorage

Transactions

Attribution

Auditing

Authorization

Dataaccess

Indexing

Encryption

Notification

Programmability

Schematization

Security

Transformation

Validation

Eventhough theDBMS itselfprovidessometype ofprogrammatic access,

thereisoftenanadditionallayerontopthatisreferredtoasaDataAccessLayer

(DAL). There are community written DALs as well as those that are

commerciallyavailable,oryoucanevenwriteyourown.TheDALwillcreate

an abstraction layer that hides the complexities of the data storage technology

and may even allow you to switch to a different backend DBMS without

affectingtheupperlayersofyourapplication.

ADALcangreatlysimplifyyouraccessbycreatinganobjectstructurethat

mightnotevenexistintheactualdatastoragesystemitself.Forexample,some

DBMSs don’tprovide schematization of data. If you need that, you can get it

through a DAL. The following image shows all the sub-layers within the data

layer:

Figure5-DataLayerwithsub-layers

1.2Datalayerdesignprocess

Before storing any data in your data layer, you need to create models for

whatthatdatashouldlooklike.Knowingwhatyouwouldliketostoreisthefirst

step to take. With that information determined, you can model what the

structural form of the data will be. This will require some thoughtful design

work to be able to organize your data in an efficient way. You will need to

diagramoutamodelforyourdatatoaidinunderstandingallthenuancesofthe

recordtypeswiththeirpropertiesandrelationships.

User interfaces come and go, but backend data systems seem to live on.

Often the data in a data layer outlives the applications that were written to

expose it. It can sometimes be hard to go back and modify your data storage

afteryouhavestartedusingit,sotakeyourtimeanddesignitforfuturegrowth.

Note: Even when the lifetime of the DBMS technology platform is finally

reached,thedatayouhavemayremainvaluable.Ifyoudecidetomigratetoa

newer DBMS technology, the data can be exported from your old DBMS and

thenbere-importedintoyournewDBMS.Mostsystemsprovidecapabilitiesto

exportandimportdatainbulk.Somesystemsevenprovideanongoingsyncing

capabilityforthetransferandtransformationofthedata.

Datalayerplanning

Makesuretoutilizeexpertsalongthewaybeforerollinganythingoutintoa

production environment. The following questions are crucial as you work

throughtheinitial datalayerdesign. Theanswers tothese questionsshouldbe

carefullydetermined.

Datalayerplanningquestionnaire:

Is the data shared? If so, how is it shared between customers,

applications,andprocesses?

Ismulti-tenantstorageok?Whatisolationofdataisnecessary?

Whatareyourdatasecurityandprivacyrequirements?

Willyoubestoringdatathatisclassifiedashighbusinessimpact

(HBI) or storing any personally identifiable information (PII)? If the

dataiscompromised,whatarethelegalramifications?

Doyouneeddataaccessrolestocontrolaccesstothedata?

Arethereanyperiodicprocessingjobsthatrunthatwillaccessthe

data?

Willparallelaccesstoasinglerecordcauseconcurrencyissues?Is

theresometypeoflockingorserializationofaccessrequired?Would

requestqueuingoroptimisticconcurrencycontrol(OCC)suffice?

Doyourequiretransactionalcapabilities?

Can writes be asynchronous or do you need immediate

synchronousacknowledgmentonawrite?

Isalagtimebetweenwriteandavailabilityofthatdataforaread

operationacceptable?

What is the service level agreement (SLA) requirement for each

CRUDoperation?

What are the access volume and rates per minute? Will there be

burstsofactivityoristheactivityevenlydistributedacrosseachday?

Whatisthesizeofthedata?Howmanyrecordsandhowlargewill

theybecome?

How many users will access the data? What will be the needed

datacapacitypercustomer?

Doyouanticipatethatthestructureofyourdatawillchange?

Doyouneedtokeeparecordofeachdataaccess,suchaskeeping

anaudittrailofaccessesandchanges?

Willyouberunningdataminingandbusinessintelligenceanalysis

overthedata?

Isthereastrictdataschemathatneedstobevalidatedagainst?

Whataretherecordtypes,contentsandrelationships?

What system dependencies are involved? How is the data

transferredandhowisitcombinedandverified?

Areyoustoringmediadatathatistypicallyinbinaryfileformsuch

asphotos,moviesetc.?

1.3IntroducingMongoDB

MongoDB is classified as a NoSQL database. This means that it is non-

relational and non-schematized. There is no central schema catalog or data

recordstructuredefinitionrequired.Thatcanexist,however,itisnotrequired.

Database records can basically be free-form. Don’t get too worried about the

unstructured nature of MongoDB records, you will learn how to set up a data

modelandhowtovalidatedatabeforeitisstoredinyourdatalayer.

MongoDBisalsoclassifiedasbeingdocument-based.Itisadocument-based

database because it conceptually stores JavaScript Object Notation (JSON)

documents. I say conceptually, because it does not actually store JSON

documents directly, but instead stores an internal binary representation.

DocumentsarestoredinBinaryJSON(BSON)form.

Document-basedstoragewithMongoDBreallyhitsasweetspotformodern

applicationneeds.Itgivesyouthebestofscalingandperformance.Becauseit

canbepurchasedasaPaaS-hostedservicethatrunsonAWSinfrastructure,itis

aneasytomanageenvironment.Youcan,ofcourse,installandrunityourself.

Note: If you look at the database offerings of AWS, you will find several

choices for data storage in the cloud, including DynamoDB, RDS/Aurora,

RedshiftandS3.Eachofferingexists tofulfilldifferentrequirements, andeach

hasitsownadvantages.MongoDBisofferedthroughMongoDBInc.

CompanyssuchasmLaborMongoDBInc.offerPaaSsolutions.TheAtlas

product from MongoDB Inc. can be installed as a service and run in cloud-

hostedservicessuch asAWS.Throughthe Atlasmanagement portal,you sign

up to use MongoDB and then your database cluster (replica set or sharded

cluster)isdeployedtocloudinfrastructureonyourbehalf.

Note: MongoDB itself is built on top of another open source component

called WiredTiger that is a data storage engine. This is something that is not

necessarytoknowabout.MongoDBisallthatyouwillinteractwithdirectly.

BenefitsofMongoDBwithAtlasPaaS

Figure 5 in the prior section showed an illustration of the data layer. This

includedthreedifferentsub-layers.Thegreatnewsisthatyougetallthreewith

MongoDB and its API access. Here are some compelling features to consider

whenevaluatingthebenefitsofMongoDB,especiallythroughtheAtlasoffering

thatusescloudinfrastructure:

Elastic storage capacity: You simply dial up and down your

storagecapacitybymovingtoahigherperformanceplanorbyadding

morecapacity.Youcan deleteinfrastructurethatis nolonger needed

and then not be charged for it anymore. You can configure your

resources through code or through a management portal. There is

practicallyunlimitedgrowthfor“pay-as-you-grow”storagecapacity.

Elasticperformance: You simply move your performance up or

downbyselectingadifferentplan.Toachievehigherthroughput,you

canpayforthehighestperformancetierandgetashardedclusterand

thehighestIOPS.

PaaS:MongoDBonAWSthroughAtlasisaPaaSoffering.Allthe

machinemanagementofsoftwareandhardwareupgradesarehandled

foryou.

APM:Capabilitiesformonitoring,alerting,scalemanagementand

backupskeepyouincontrol.

Features: Besides creating, reading, updating and deleting

documents, there are many features to use under the right

circumstance. Features such as views, indexing, replication, change

streams, sharding, aggregation, capped collections, TTL indexes,

accesscontrol,encryption,ACIDtransactionsandmuchmore.

Auto-Replication: Data is automatically stored in redundant

copies of the database. The replications are there for your safety, to

ensure availability through failover. You can also set up automatic

backupstohappen.

Security: MongoDB has an audit log to track all database

operations. For added security, you can have an SSL connection to

your database. MongoDB also has an encrypted storage engine

availabletoprotectdata.

TheAtlasMongoDBofferingonAWSisreadytomeetallyourbusinessand

customerneeds.Ithasagreatfeaturesetandisincreasingincapabilitiesallthe

time.

Tryitout

Ifyoureallywantedto,youcoulddownloadMongoDBandrunitonyour

localmachine.Youcouldalsodotheworktosetupacloud-hostedVMorusea

DockercontainerwithMongoDB.MypreferenceistouseMongoDBasacloud-

hostedPaaSsolution.

MongoDBInc.makesiteasytosignupforaplanthatthenhostsMongoDB

onyour choiceofAmazon AWS, MicrosoftAzure, orGoogle CloudServices.

ThisistheapproachIhaveusedforthisbook.Youwouldhavetodoyourown

researchifyouwanttodepartfromthat.Therestofthisbookiswrittenwiththis

approachinmind.

It takes just a few clicks to and be up and running with MongoDB as a

hosted service in AWS. You don’t need to worry about the daily details of

managingthesoftwareupdatesandmachinehardwaremaintenancetokeepitup

andrunning.Thereisnoneedtowasteanytimedealingwithhardwarefailures,

suchasthereplacementofdrivesornetworkcards.

Youenjoythebenefitofhavingmachinereplication,scaling,load-balancing,

failover, and backup. With the scaling of MongoDB, you only pay for the

storageand performance you desire. Youpay for what you use and can easily

scaledownwhenyounolongerneedasmuchdatastorageorperformance.

You are free to concentrate on the aspects of your application that deliver

value to your customer and increase ROI for your company. There are many

benefitswithPaaScloudinfrastructure.

Payforperformance

As previously mentioned, you only pay for what you need. As with any

cloud infrastructure, if you have an immediate need, you can pay for higher

performance machines to run MongoDB. Thankfully, there is a free option

throughAtlasforyourinitialinvestigations.

Youcanalsopayforyourhigherscalingcapabilitybypurchasingaplanthat

comeswithahigherstorageallotmentandisconfiguredwithmoremachinesfor

high-availabilityandauto-failover.Ifyouneedtoscaleuptomorestorage,you

cankeepaddingdatabaseclustersasneededandspreadyourdataacrossmore

machineswithsharding.

Youalsohavetheflexibilitytopayformorethanoneplanatthesametime

andputdataonthemoreexpensiveplanthatneedsthehighestthroughput,while

otherdatacanbeonthelessexpensiveplanofstorage.Youcanchangethings

liketheinstancesize,replicationfactorandshardingscaleasneededatanytime.

Insomecases,aconfigurationchangewillstillrequiredowntime,suchasa

change in the instance size. This downtime would happen while the primary

serverisbeingmigrated.Butthistimeshouldbeunderaminute.SeetheAtlas

documentationformoredetails.

MongoDBstructure

WithyourAtlasaccount,youcancreateoneormoredatabaseclusters.These

clusters then have databases that serve as the containers for what are called

collections.Databasesholdcollectionsandtheyalsoholdyourindexesanduser

accounts.Collections,however,arewhatcontainyourdataintheformofBSON

documents.

Each database can have a set of users with specific permissions. You can

controltheusersofyourdatabaseandgiveindividualsreadaccessandalsowrite

access.ThisdoesnotmeanthateveryonethatconnectstoMongoDBthroughthe

NewsWatcherapp needs theirown useraccount. Generalprogrammatic access

happensthroughyourservicelayer.Specificdataaccessiscontrolledthrougha

middle-tier login mechanism. That information will be covered later in this

book.

Note:Youcancreateuseraccountsattheclusterlevelthatworkacrossall

databasescreatedonthatcluster.Anaccountiscreatedwhenyoufirstcreatethe

cluster,foradminprivileges.

Thefollowingillustrationisanoverall visualrepresentationthatshowsthe

differentresourcesthatarepartoftheMongoDBmanagedplatformserviceand

howtheyrelatetoeachother.Theimportantthingtounderstandisthatyoucan

havemultipledatabases,witheachdatabasehavingmultiplecollectionsinit.

Figure6-MongoDBresources

1.4TheMongoDBCollection

A collection is a container for storing documents and is the main resource

withwhichinteractionshappenprogrammatically.YoucanwriteJavaScriptcode

inyourNode.jsservicelayerforoperatingonthedatainthecollection.

Collections are a convenient way to separate out data in a MongoDB

database. They collectively share whatever the limit of total storage is for the

planyouarepayingfor.

Towritedataintoacollection,youwillwanttouseoneoftheAPIsavailable

todothat.YoucanalsowritedatausingtheMongoDBcommandlinetool,or

theCompassUIapplicationthatcanbeinstalledonyourmachine.Youcanadd

any documents by hand. Most likely, you will want to use one of the code

languageAPIsavailable.

1.5TheMongoDBDocument

Everydatabaseeverinventedisfundamentallydesignedtostoredataasaset

of records. Each individual record contains information that is useful for later

retrieval.Forexample,inarelationaldatabase,therecordisintheformofarow

in a table. For example, you might have a table of customers where each row

represents a single customer with their name, age, and email as part of their

record.InAWSS3,arecordtakestheformofdatawithitsdescriptivemetadata.

InMongoDB,asinglerecordisrepresentedbyadocumentinacollection.

The following image illustrates how you can have multiple collections, with

each collection containing multiple documents. Each rectangle is a document

below:

Figure7-ExampleMongoDBcollectionswithdocuments

Documents in this example, are represented as JSON with top-level curly

braces enclosing each document. This is how they are shown textually, even

thoughtheyreallydonotappearthatwayinthedatabaseitself.

Thissimpleconceptofstoringdocumentsinadocument-baseddatabasehas

many advantages. JSON is easy to formulate and consume in code. Another

benefitisthatitcancontaincomplexhierarchicaldatainembeddedstructures.It

alsosupportsarrays.

Datatypesinadocument

TheprimitivedatatypessupportedinaMongoDBJSONdocumentarethe

sameonesthatareavailableinJSON.Theseare:

Array

Boolean

Null

Number

Object

String

Inaddition,thereisanextendedsyntaxinJSONthatcantaketheaugmented

BSONdatatypesandpreservethem.SomeoftheaddeddatatypesofBSONare

Date, Binary data, 64-bit integer and many others. You will eventually be

accessing MongoDB through code and will be using JavaScript objects with

theirsupporteddatatypes.

Iwillnowgiveyouafewtipsforstoringmediadataandcurrencyvalues,as

therearesomeinterestingtechniquestoutilize.

Oneoftheseissuesiswiththestorageofbinarydata,especiallymediadata

such as photos and movies. Media data cannot realistically be inserted into a

JSONfile.YoucouldusetheBSONbinarydatatype,butthemainproblemyou

would encounter would be that the maximum BSON document size is 16

megabytes.Amoviewouldsimplynotfitintoonesingledocument.Itisbestto

storebinarydatainaseparatestoragesystem,suchasAWSS3,andreferenceit

fromaMongoDBdocument.

Another issue is the representation of currency values, such as US dollars

and cents. The problem comes when you attempt to perform floating point

precision storage and arithmetic operations, such as might be required for

investment calculations. If Node.js were to calculate and print 0.1 + 0.02, the

result would be 0.12000000000000001. You are not even safe in using the

MongoDB 64-bit floating point datatype. This is due to the rounding errors

inherentinCPUsbecauseofhowtheyrepresentfloatingpointnumbers.

One solution is to multiply all your monetary values by a constant scaling

factor. Forexample, if you weregoing to store the valueof $1.99, don’tstore

1.99,butinsteadmultiplythatby1000andstoreanintegervalueof1990.Then

youcanperformmathonthoseintegervaluesandconvertbackfordisplaywhen

needed.

TheJSONdocument

Onedistinguishingcharacteristicofadocument-baseddatabaseisthatitcan

storecomplexhierarchicalobjectswithoutanypredefinedschema.This means

thateverydocumentinacollectioncouldhaveadifferentstructure.Thiscould

beconsideredbothanadvantageandadisadvantage.

Therearemanyadvantages tohaving aschema-less database.The obvious

oneisthatyoudon’tneedtospecifytheJSONformatinadvance.Thismeans

that there is no need to specify datatypes or have restrictions on them. It is

simplyamatterofcreatingtheJSONwiththename/valuepairsthatyoudesire

andtheninsertingthatintoacollection.HereisasimpleJSONdocument:

{

"_id":ObjectID("59612a3dc17c5416d0a33041"),

"myNumber":99,

"myString":"Hithere",

"myBool":true

}

Note:Irefertothename/valuepairsintheJSONDocumentas“properties”,

sinceitssyntaxis soclose tothesyntaxof theproperty inaJavaScript object

literal.TheMongoDBonlinedocumentation,however,usesthetermfield/value

forthepairs.

IfyouopenoftheMongoDBInc.Compassappandlookatsomedocuments,

youcansee eachinthe UIwithall theirpropertiestoexplore. Inthischapter,

documentswillbepresentedinthe textualJSONformsinceyoucan visualize

thembetterthatway,andifneeded,importthemintoacollectioninthatform.

Once you get to the chapter on Node.js and are using the API to

programmaticallyinteractwithMongoDB,youwillseeJavaScriptcodewiththe

objectliteralsyntaxused.Justbeawareofthatswitch.Asanexample,theobject

literal syntax equivalent of the prior JSON example would be as follows

(MongoDBwillprovidetheidpropertyuponinsertion):

varmydoc={

myNumber:99,

myString:"Hithere",

myBool:true

};

Note: There is one restriction on JSON documents to make them work in

MongoDB.Therestrictionisthatthepropertynamesinasingledocumentmust

notbeduplicated.Thismakessensebecause,ifyouweretoqueryandaskfora

given property and it existed twice, that would be a little confusing. The core

MongoDBstoragesystemthatstores BSONdoesnotmake thisrestriction,but

theNode.jsmoduleAPIthataccessesitrequiresit.

Anexample

Let’s pretendyou are runninga business thatsells books overthe internet.

Thefollowingdocumentrepresentswhatyoumightwanttouseinacollection

thatholdscustomersofyouronlinebookstore.Eachcustomerdocumentwould

hold information associated with that customer. You would want to store

information about what books each customer had purchased. The book order

datafor the customer is embeddedin their document. Youwould also wantto

storetheirpersonalcontactinformation.Allofthiscanbeplacedinonesingle

documentasfollows:

//CustomerDocument

{

"_id":"77",

"name":"JoeSchmoe",

"age":27,

"email":"js@live.com",

"address":{

"street":"21MainStreet",

"city":"EmeraldCity",

"state":"KS",

"postalCode":"10021-3100"

"booksPurchased":[

{

"title":"AgileProjectManagementwithKanban",

"ISBN10":"0735698953",

"author":"EricBrechner",

"pages":160,

"publicationDate":"20150326",

"category":"SoftwareEngineering"

{

"title":"TheMerriam-WebsterDictionary",

"ISBN10":"087779930X",

"author":"Merriam-Webster",

"pages":939,

"publicationDate":"20040701"

"category":"English"

}

]

}

Itisimportanttorememberthatasingledocumentshouldonlycontainthe

informationforthatoneuniquerecord.Thismeansthatyouwouldnotwantto

havetwocustomersinasingledocument.Thatwouldgetconfusing.

Aspreviouslymentioned,documentsdonotneedtobecompletelyuniform.

This means that one document can contain properties that another document

mightnoteverhave,andbothcanexistinthesamecollection.Forexample,if

youhadacollectionthatcontainedproductsforthefictitiousbookstore,itwould

obviouslyhavebooksinit.Itmightalsohavemagazines,maps,andpuzzlesinit

aswell.Thedetailsoftheseproductswouldneedtobesomewhatdifferent.

Documentsinacollectionmighteachhavesomecommonpropertiessuchas

price,title,description,andweight.Foreachproducttype,therewouldbesome

uniqueproperties.Apuzzle,forexample,mighthaveapropertythatstateswhat

therecommendedageisforthatpuzzle.

The following example shows a collection of documents that have both

common and unique properties. Note how the document for a book differs

slightly from the document for a puzzle, yet both can exist in the same

collection.

//Bookstoreproducts

{

"type":"BOOK",

"title":"AgileProjectManagementwithKanban",

"ISBN10":"0735698953",

"author":"EricBrechner",

"pages":160,

"publicationDate":"20150326",

"category":"SoftwareEngineering"

{

"type":"BOOK",

"title":"TheMerriam-WebsterDictionary",

"ISBN10":"087779930X",

"author":"Merriam-Webster",

"pages":939,

"publicationDate":"20040701"

"category":"English"

{

"type":"PUZZLE",

"title":"Balloonsinsky",

"company":"ZipZapToys",

"age":"3-5yearsold"

}

Just because documents can contain heterogeneous content does not mean

that you always want to have collections set up that way. You may decide to

have all your collections contain uniformly structured documents. I will later

describecircumstancesthatwillhelpyoumakethesetypeofdesigndecisions.

Note:Iwouldadviseyoutostayawayfrommultidimensionalproperties(an

array of arrays). Otherwise, you may find yourself looking through your data

andnotrememberinghowyouhadsetitup.

Acommonpropertyofalldocuments

Each document has an _id property. When a document is created, it will

alwayshaveauniqueidassociatedwithitsothatitcanbeidentified.Thisserves

astheprimarykey.Youcansetthevalueyourself.Ifyoudon’t,MongoDBwill

setavalueforyouoftypeObjectId.Ifyousetityourself,itcanbeofanytype

other than an array. It must, however, be unique across all documents in that

collection.

Ifyoueverneedtodirectlyaccessadocument,youcanaccessadocument

usingthe_idasthequickestwaytoqueryforit.

Referencingexternaldata

Sofar,youhaveseenthatMongoDBallowsforstorageofJSONdocuments.

Asyouknow,thebasicdatatypesavailablewithJSONcanbeusedtorepresent

lotsofinformationyouwishtostore.

Aswasmentioned,theonethingthatJSONdatatypesarenotsuitedfor,is

the representation of large binary data. For example, you would not really be

abletostorelargemediacontentsuchasphotos,music,orvideoinadocument.

Toovercomethislimitation,youcancreateapropertythatisareferenceto

wheretheactualmediacontentisexternallystored.YouwoulduseaUniversal

ResourceIndicator(URI)toindicateitslocation,andcreateaparallelproperty

thatdescribesthetypeofdataitconsistsof:text,image,binarydata,etc.That

way,yourcodecaninterpretitcorrectly.

Theexternallyreferenceddatacanbeanydatayouwouldlike.Itisuptoyou

tosetthetypeandthentreatitassuchwhenyouretrieveit.Besuretohandle

anyneededdeletionofyourexternaldataifitissupposedtobecleanedupwhen

documentsreferencingitaredeleted.

Note:ThischapterstatedthatMongoDB,asadocument-baseddatabase,is

notrequiredtoenforceanyschema.MongoDBdoeshaveacapabilitywhereyou

canspecifywhatpropertiesshouldexistandwhatrestrictionsshouldbeonthem

fordocuments.Idonotutilizethiscapabilityinthisbook.Idolatermentionthat

youcanuseaNode.jsmodulesuchasMongoosetoschematizeyourdataand/or

usejoitovalidateyourschema.ThevalidationMongoDBperformsisalsonot

as strict a definition as you have with a relational database. For example,

relationshipkeyspecificationsareavailablewitharelationaldatabasewiththe

abilitytoalsoenforcewhatisreferredtoasreferentialintegrity.

Chapter2:DataModeling

A data model defines the structure of records that are to be stored in a

database.Thisincludesinformationonhowthedifferenttypesofrecordsrelate

toeachother.

Justbecauseadocument-baseddatabaseisnotarelationaldatabase,doesnot

mean that it doesn’t have structure or relationships between document types.

Specifying a data model in MongoDB consists of specifying what the JSON

documentsare,whichcollectionstheyexistin,andhowtheyrelatetoeachother.

Iwillinstructyouonhowtousevalidationcodetomakesureanydatagoingin

conformstoexpectationsthatconformtoyourdatamodel.

Sketchingout yourdata modelwill be animportant stepin the creationof

yourdata layer. Itis helpful todesign yourdocument structures inadvance so

thatyoucanlookatallaspectsof yourdata.Youcan matchyourdatastorage

needs against the characteristics that a MongoDB database offers and do your

datamodelingaccordingly.

Tovisualizeyourdatamodel,youcanusewhateverdiagramformatortool

youlike.Youcanevenscribbleitoutonapieceofpaper,althoughyoumight

wantotkeepitinelectronicformforeasierediting.Ultimately,withMongoDB,

theJSONformatiswhatyouwouldneedtospecify.Let’snowcoversomeofthe

bigdesigndecisionsthatneedtobemadewithadocument-basedDBMS.

2.1ReferencingorEmbeddingData

Recordsinarelationaldatabaseeachcontainkeystoidentifythemandtoact

asreferencestoeachother.ArelationalDBMShasmechanismstospecifyand

enforcetheintegrityofthesereferencestosomeextent.

You may have heard the term “normalization” used in the context of

relational database designs. Normalization requires the separating out of data

into different record types and relating them to each other. For example, you

mighthavetwotypesofrecordsinyourdatabase,onethatrepresentspeopleand

anotherthatrepresentsthepetsthatareownedbythepeople.Therewouldbea

key to relate an owner to one or more of their pets. The following illustration

showsthisnormalizedstructure:

Figure8-Normalizedstructure

InarelationalDBMS,yousetupthekeystorelatetherecordtypestoone

another.TherelationalDBMSprovidesfunctionalitytobeabletoqueryandjoin

therecordstogether.Thismeansyoucanquerythedatabasetoreturnaperson

andallthepetstheypossess.Thisisdonethroughasinglequeryoperation.

Let’s now bring this conversation into the MongoDB world. Following a

normalized model with MongoDB, you would create separate documents for

eachpersonandeachpetallinthesamecollection.Keepinmindthatthereisno

conceptofaschemaforacollectionrequired,andnoconceptofrelationalkeys

orcross-documentjoinqueries.Youwouldneedtodesignyourownproperties

oneachdocumentinordertorelatepetstotheirowners.Withpeopleandtheir

petsseparatedoutacrossMongoDBdocuments,youhaveachievedanormalized

databasemodel.

Anormalizedmodel,however,isnotnecessarilythebestwayofstoringdata

in a document-based database. You really want to think more about how to

denormalizeyourdata.Denormalizeddatameansthateverythingisallbundled

together.

To denormalize, you bundle pets together with a person. You do this by

simply placing pets as an array property that is embedded in each person

document. Imagine stuffingall the pets in the pocket of the person. This way,

people and their pets are always found together. In this way, there are no

relational links required. You also do not need a join operation if you

denormalizeyour data. In fact, the classic relationaljoin operation is not even

supportedinMongoDB.Thefollowingvisualizationshowshowthepersonand

petstraveltogether:

Figure9-Denormalizedstructure

Aswasmentioned,document-baseddatabaseslikeMongoDBdonotsupport

joinsofrecordssuchascross-tablejoinsdonewitharelationaldatabase.Thisis

becausecross-documentjoinsdonotmakesenseindocument-baseddatabases.

Thismeansthatyouneedtogetcomfortablewithkeepingyourdatabasedesigns

denormalized.Denormalizationismoreefficientandworkswellinadocument-

based database. What you do, is make use of the array and embedded object

propertiesinyourJSON.

In the previous chapter, I showed you a JSON document representing a

customer of an online bookstore company. What was shown was actually a

denormalized data pattern. Each customer document contained information

aboutthecustomer,butitalsocontainedinformationabouteachbooktheyhad

purchased.Youwillendupwithlargerandmorecomplexdocumentswhenyou

denormalizeyourdata.

Even if you end up with large, complex documents in MongoDB, the

queryingcapabilitysupportsretrievalofjusttheportionsofthedocumentsthat

youneed. Forexample, if youkept petsembedded in aperson document, you

candoaquerytojustreturnthepetsofagivenowner.Youdonotneedtoreturn

thecompletepersondocumentifyoudonotwantto.Perhapsyoujustwantto

knowthenamesofallthepetsanddonotwanttoretrieveanythingelseabout

thepetortheperson.

Insummary,itcanbestatedthatdatabasenormalizationhasyouseparateout

yourdataintodifferentdistinctrecordtypesandhasyousetupkeysasreference

linksbetweenthem.On theotherhand,denormalization,with document-based

databases, has you keep as much data bundled together as possible by

embeddingdatathatisrelated.

2.2WhentouseReferencing

Itisobviouslynotgoingtoworkwellifyoualwaysembedallyourdataand

end up with huge, complicated documents. You need to make decisions as to

what properties will be embedded in a document and what would make more

sensetopulloutintoseparatedocumenttypestoreference.

Youcankeepallofyourdifferentdocumenttypesinonecollection.There

are,however,reasonswhydifferentdocumenttypesshouldbekeptinseparate

collectionsorevenseparatedatabases.Iwillcovermoreonthattopiclater.

Performanceimplications

Your data model design will have an impact on the performance of all

database operations. For example, the performance of reads and writes could

improve if you spread data out into separate documents. Doing this, results in

decreaseddatatransferamountsandmoreefficientdocumentupdatesaswell.

The downside of referencing data from one document to another is that it

willbeslowerifyouneedtocombinedataandpresentittogetheratanypointin

time.Imaginethecoderequiredtopiecetogetherapersonwiththeirpetsifpet

documentswerekeptseparate.

Let’s goback to theonline bookstore example.Imagine thatyou have two

documenttypes:customersandbooks.Withthesedocuments,youkeeptrackof

allthebookpurchasesofindividualcustomers.IfyoukeeponlythebookIDsin

the customer document, certain queries would run slower, such as listing a

customerwiththebooktitlestheyordered.Foreachcustomer,youhavetolook

through the booksPurchased array property and then do individual fetches of

dataforeachbookidlisted.Hereishowthisnormalizedmodellooks:

//Customerdocuments

{

"_id":"77",

"type":"CUSTOMER_TYPE",

"name":"JoeSchmoe",

"age":27,

"email":"js@gmail.com",

"address":{

"street":"21MainStreet",

"city":"EmeraldCity",

"state":"KS",

"postalCode":"10021-3100"

"booksPurchased":["0735698953","087779930X"]

}

//BookDocuments

{

"_id":"7865",

"type":"BOOK_TYPE",

"ISBN10":"0735698953",

"title":"AgileProjectManagementwithKanban",

"author":"EricBrechner",

"pages":160,

"bookReviews":[

{

"reviewer":"JoeSchmoe",

"comments":"WishIhadthisyearsago!",

"rating":4

{

"reviewer":"JaneDoe",

"comments":"Foreveraclassic.",

"rating":5

}

]

}

{

"_id":"7866",

"type":"BOOK_TYPE",

"ISBN10":"087779930X",

"title":"TheMerriam-WebsterDictionary",

"author":"Merriam-Webster",

"pages":939,

"bookReviews":[]

}

Imaginethatyouwanttoqueryandfindallbooksthatareover500pagesin

lengthandthatwerepurchasedbyaparticularcustomer,JoeSchmoe.Todothis,

thequerymustfirstsearchthecollectiontofindtheJoeSchmoedocumentand

then,foreverybookIDlistedintheJoeSchmoedocument,lookupthatbook

documentandseeifthepagespropertyvalueisover500,andthenfinallyreturn

thosedocuments.Thisinvolvesmorethanonedocument.

Inrelationaldatabases,thiscanbedone inasinglequery.InMongoDB, it

mustbedoneinseparatequeriesandinvolvescodetopieceeverythingtogether

ifneeded.

Thisisnotnecessarilyabadthingandisjustpartofwhatyouneedtodoina

document-based database when you separate out data into related documents.

But then again, remember that denormalization relieves you from doing these

joinoperationsacrossdocuments.Youwilllaterseethatthereisacompromise

thatcanbemadebetweenthetwooptions.

Whentoreference

Let’s cover the basic scenarios that would cause you to split data across

documents with a normalized design that uses references. The following are

someofthecommonreasonstodothis:

Thedataisrarelyusedinqueries:

Ifyou find thatyou rarely reference certainproperties in adocument,

thenyoumightseethisasasignthatthosepropertiesbelonginaseparate

document,oreveninaseparatecollectionthatcanbereferencedasneeded.

Forexample,theaddresspropertyinthecustomerdocumentcouldbetaken

out and placed into a separate document if it is decided that it is rarely

needed.Justrememberthatyouaregainingthefasterinteractionwiththe

primary data at the expense of slower data retrieval processing to join it

togetherlater.

Thedataiscommonacrossdocuments:

The book detail is certainly data that would be duplicated across

customers. If you had thousands of customers that each had a Merriam-

WebsterDictionary,itwouldnotbenecessarytokeepduplicatingallofthe

datafor thatbook. Withduplicated data,one bigproblem is ifone ofthe

propertiesofabookneedstobealtered,itwouldrequirealteringitacross

alltheduplicatesinsteadofinonecentraldocument.Inthiscase,itwould

be wise to keep the properties in a separate book document if they are

sharedandupdatedfrequentlyandmakeareferencetothem.

Thedatahasmutualorcross-referencerelationship:

The information in the bookReviews property originates from the

customers that submit the reviews, but each review is also specific to a

singlebook.Thequestionarises—shouldthebookreviewsexistwiththe

book being reviewed, or with the customer giving them? You might not

wanttoupdatetheindividualbook documentandkeepaddingto itevery

time a new customer adds a review. Nor do you want to fetch all of the

bookreviewseverytimeyoufetchthedocumentforabook.Thisiswhere

you can make the case for the book reviews for each book to be in their

ownseparate document withan array property orhave each in individual

documents and then referenced by both the customer and the book

documents.

Thedatawillgrowverylarge:

An array property might grow to have a large number of entries. An

arraypropertyofadocumentcannotgrowunbounded.Rememberthatthere

isa16-megabytelimitonthesizeofanindividualdocumentinMongoDB.

Youwouldhavetosetupseveralrelateddocumentsthateachhadanarray

thatcomprisedsectionsofthetotalset.

2.3ReferenceRelationshipPatterns

AtonepointIshowedyouacustomerdocumentthathadalistthatcontained

theidsofthebooksapersonhadpurchased.Thisdesignkeepsthedocuments

fortheactualbookdetailsseparatedout.Thisisoneofseveralpatternsyoucan

findusefulforreferencingdata.

Iwillstressagainthough,thatthisistotallyunderyourcontroltoimplement.

MongoDB does not provide any built-in recognition of relationships. It is

importanttorealizethatMongoDBwillnotverifythereferentialintegrityofthe

dataasarelationalDBMSmightdo.Itisuptoyoutomanagethat.Ifagiven

bookisdeletedfromacollection,youalsoneedtodeleteallthereferencestoit.

Forexample,ifacustomerdocumenthadabooksPurchasedarraypropertywith

anentryinitwithanidof5777,thereisnothinginMongoDBthatisverifying

thatabookdocumentactuallyexistswithanidof5777.

Herearesomecommonpatternsthatcanbeusedforreferencingdataacross

documentsinMongoDB:

One-to-One: This is where you would separate out a piece of

infrequently accessed data. For example, you could pull out the

address information of a customer and put it into its own document.

Thisdiagramillustratesaone-to-onepattern:

Figure10-One-to-onerelationshippattern

Many-to-One:This is where one document might be referenced

bymanyotherdocuments.Thisistheexampleyouhavealreadyseen

where a single book can be referenced by multiple customers. Each

entryinthebooksPurchasedarrayhasanISBN10IDtoreferencethe

bookdocumentwith.Thisdiagramillustratesamany-to-onepattern:

Figure11-Many-to-onerelationshippattern

Many-to-ManyAssociation:Thisiswhereyouhavetwoseparate

documenttypeswhereneitherreferencestheother,butinsteadthereis

a third document type that can tie the two together through an

association.Thisthirddocumentcanalsocontaininformationrelevant

to that association. For example, you might have a document for

purchasesthatrecordsthedetailsofthetransactionofabookpurchase.

Thus, you have many purchase documents that reference many

customer and book documents. This diagram illustrates a many-to-

manyassociationpattern:

Figure12-Many-to-manyassociationpattern

Many-to-manyrelationshipscouldhavedocumentsreferencingeachotherin

acircularmanner.Trynottogetthatcomplicated,asitpresentsmanydifficult

dataintegrityissuesandcomplicatestheservicelayerimplementation.

Note:Itisusefultocreateavisualdiagramofyourdocumenttypesasyou

gothrough yourdatamodel designprocess. Youarethen ableto moreclearly

see the structure and relationships. Several diagram standards have been

createdovertheyearstovisuallyrepresentobjectsinobject-orientedlanguages

and for representing records in databases. All you really need is a simplified

visual representation of JSON objects in your data model. You can draw a

rectangle shape for each document type and list all the properties inside as I

haveshowninthepreviousillustrations.IhavechosentohavetheIDproperty

existabovealine.Thosepropertiesbelowthelinearetherestofthedocument

properties. I have chosen to use a greater than sign to show sub-object

properties.Anarrayofobjectsisshowninparenthesisanditisunderstoodthere

wouldbezeroormoreoftheseinanactualinstanceofthedocument.Youmight

alsowanttolistthedatatypeofeachpropertyouttotherightofthename.

2.4AHybridApproach

Youare now ready to learn about a hybrid approach thatcan give you the

bestofbothtechniquesforbothreferencingaswellasembeddingdata.Whynot

combinethetwotechniquesinasortofcompromise?Forexample,ifyoufind

thatyouoftenneedtolistthetitlesofthebooksthatacustomerowns,youcan

duplicateaportionofthatinformationacrossdocuments.Thismeansyoustore

thebookidandthetitle,eventhoughitisduplicatedinformation.Thefollowing

isanexampleofwhatthedocumentswouldlooklike:

//CustomerDocument

{

"_id":"77",

"type":"CUSTOMER_TYPE",

"name":"JoeSchmoe",

...

"booksPurchased":[

{

"ISBN10":"0735698953",

"title":"AgileProjectManagementwithKanban"

{

"ISBN10":"087779930X",

"title":"TheMerriam-WebsterDictionary"

}

...

//BookDocument

{

"_id":"77",

"type":"BOOK_TYPE",

"ISBN10":"0735698953",

"title":"AgileProjectManagementwithKanban",

"author":"EricBrechner",

"pages":160,

"publicationDate":"20150326",

"category":"SoftwareEngineering"

You can see that the compromise was to keep the title duplicated across

documenttypes,asthetitleisfrequentlyneeded.Therestofthepropertiesare

keptseparatedout.Beawarethatyouwouldwanttosynchronizechangestoany

duplicatedpropertiesthatexisted.Forexample,ifyouneedtomakeachangeto

thetitle property in the book document for a given book, you would want to

havesomebackgroundprocessthatwouldqueryalloccurrencesofthatbookin

thecustomerdocumentsandupdatethetitleinthose.

2.5DifferentiatingDocumentTypes

You might have several different document types existing in a single

collection.Storingmultipledocumenttypesinonesinglecollectionallowsyou

to use a single API connection in your code. API connections are tied to a

databaseandforasinglecollection.

With all document types in the same collection, you would need a way of

differentiatingthemfromeachothersothatqueriescanfindeachspecifictype.

Onewaytosolvethisistoincludeatypepropertyineachdocument.

Let me use the previous example of documents for persons and for pets.

Person documents would be declared with: "type":"PERSON_TYPE" and pet

documentswouldbedeclaredwith"type":"PET_TYPE".Forexample:

//PersonandPettypesinsameCollection

{

"type":"PERSON_TYPE",

"_id":1,

"name":"Ian",

"Petclubmembership":true,

"pets":[12,24],

{

"type":"PET_TYPE",

"_id":12,

"name":"Kirby",

"breed":"Cavalierdog",

{

"type":"PET_TYPE",

"_id":24,

"name":"Kaitlyn",

"breed":"Siamesecat",

}

Thisallows a query to narrow down results to just returningthe document

typeyouwantandthenyoucanaddinwhateverfurthercriteriayouarelooking

for.

2.6RunningOutofSpaceinaDatabase

Thereare ways to deal withcases where the number or sizeof documents

becomes a problem. Think about what would happen if you keep adding

documents to a collection that existed on a single SSD (Solid State Disk)?

Obviously,atsomepoint,youaregoingtoreachthestoragespacelimitthatis

setforasingledatabaseyouarepayingfor.Thisdoesnotnecessarilymeanthat

youneed to separateout and referencedata across documenttypes inseparate

collections. You can still keep data embedded and grow the number of

documentsindefinitely.

Themethodforachievingstoragecapacityscalingisthroughwhatiscalled

sharding.Thismeansyourcollectionismoreofalogicalconceptandisactually

spread across multiple SSDs. The unit of scaling for MongoDB is called a

replicaset.Agiven document,however,mustonlybefound inonereplicaset

(shard)oftheshardedcluster.ThenicethingisthatMongoDBhidesthatfrom

you,andyourqueryorupdatedoesnotevenrealizewhatisgoingon.

Forcustomerdocuments,youcouldhavethingssetuptodistributecustomer

documents out across different replica set storage. In a later chapter, I will

discussthistypeofdatapartitioning.Don’tworryifyoudonotfullyunderstand

thisconceptjustyet.Youmightneverneedtoimplementshardinganyway,only

ifyourunintoreallylargeamountsofdataandneedtomaintainfastreadand

writetimes.

2.7AccessControl

There are several ways to interact with MongoDB documents. One is

through the Atlas management web portal. Another is through the Compass

application, and another is through API access, such as in Node.js JavaScript

code.Ofcourse,thereisalsotheMongoshell,butthatisnotsomethingthatwill

beneededforthepurposesofwhatthisbookisteaching.Youwilllearnjustthe

bareminimumtousethemongoshell,suchasforimportingdocumentsinbulk.

If you want to learn more about the Mongo shell, please refer to MongoDB’s

documentation.

FromtheAtlasmanagementportal,auseraccountcanbeaddedtoprovide

authenticationforthatuser.Youcangiveauserread-onlyprivileges,ifyouneed

thatrestrictioninplace.Accountadministrationthroughtheportalisnotcentral

to the topic of this book, so if you need more information, please refer to the

Atlasmanagementportaldocumentation.

As mentioned, the primary type of access discussed in this book is done

programmatically through a MongoDB API, such as one that is provided for

Node.jsdevelopers.ThedetailsofthiswillbeexplainedintheNode.jssection

laterinthisbook.

You will later see how the NewsWatcher sample application has a UI that

goes through the middle-tier service layer where interaction with the database

takes place. The middle tier will then be able to authenticate on behalf of the

user and access the database on their behalf and can restrict access to just

documentsthataparticularuserisallowedtosee.

Chapter3:QueryingforDocuments

Youmayhaveheard ofSQL(StructuredQuery Language)asthelanguage

usedtoqueryforrecordsinrelationaldatabases.UsingSQL,yousubmitqueries

to your DBMS and receive back the resulting records that match. MongoDB

supportsquerying,butitdoesnotuseSQL.Instead,ithasitsownquerysyntax.

The lowest level interface to MongoDB has a means of interaction to

accomplish operations such as queries. Everything at the lowest level is done

through a TCP/IP connection that has a well-defined wire protocol for the

operationsthatitsupports.Therearearoundnineoperationsyoucanmakewith

thisprotocol.Don’tworryaboutunderstandingthis,allyouneedtoknowisthat

otherpeoplehavedonetheworktoabstractawaythecomplexitiesofusingthis

protocol by creating specific language drivers you can use. This book will be

concerned with using the Node.js JavaScript driver. The following diagram

showstheoverallaccesslayers:

Figure13-MongoDBaccessabstractionthroughAPIs

In the service layer part of this book, I will show you how to use the

MongoDBNode.jsdriverforoperationsonthedatasuchascreate,read,update,

and delete (CRUD). This current chapter will only cover the specifics of the

syntaxforqueryoperationsingeneral.

Regardless of which of the CRUD operations you perform, you need to

specify your query criteria as part of that request. You can explore this topic

now,becauseyoudon’tneedtowriteanycodetotryoutyourqueries.Youcan

usetheCompassapplicationtotrythemoutandexperimentwiththesyntaxas

youlike.

Exampledocuments

CarefullyreviewtheexampleJSONdocumentsshownbelow.Theywillbe

used with examples showing how to construct your queries. You can imagine

thesedocumentsbeingusedbyanonlinebookstore.Therewouldobviouslybea

lotmoredataavailablethanwhatisinthisexample:

//CustomerDocuments

{

"_id":"77",

"type":"CUSTOMER_TYPE",

"name":"JoeSchmoe",

"age":27,

"email":"js@gmail.com",

"address":{

"street":"21MainStreet",

"city":"EmeraldCity",

"state":"KS",

"postalCode":"10021-3100"

"rewardsPoints":99,

"booksPurchased":[

{

"id":"1098",

"title":"AgileProjectManagementwithKanban"

{

"id":"1099",

"title":"TheMerriam-WebsterDictionary"

}

]

},{

"_id":"78",

"type":"CUSTOMER_TYPE",

"name":"JaneDoe",

"age":37,

"email":"jd@gmail.com",

"address":{

"street":"100SBridgerBlvd",

"city":"Paradise",

"state":"UT",

"postalCode":"84328"

"rewardsPoints":0

}

//BookDocuments

{

"_id":"1098",

"type":"BOOK_TYPE",

"title":"AgileProjectManagementwithKanban",

"ISBN10":"0735698953",

"author":"EricBrechner",

"pages":160,

"format":"Paperback",

"price":27.66,

"publicationDate":"20150326",

"category":"SoftwareEngineering",

"bookReviews":[

{

"reviewer":"JoeSchmoe",

"date":"20140321",

"comments":"WishIhadthisyearsago!",

"rating":4

},{

"reviewer":"JaneDoe",

"date":"20150923",

"comments":"Foreveraclassic.",

"rating":5

}]

},{

"_id":"1099",

"type":"BOOK_TYPE",

"title":"TheMerriam-WebsterDictionary",

"ISBN10":"087779930X",

"author":"Merriam-Webster",

"pages":939,

"format":"Paperback",

"price":11.26,

"publicationDate":"20040701",

"category":"English",

"bookReviews":[

{

"reviewer":"JoeSchmoe",

"date":"20100101",

"comments":"Aterrificvolumetokeephandy.",

"rating":4

},{

"reviewer":"JaneDoe",

"date":"20120817",

"comments":"Wishitcameinanaudiobookformat.",

"rating":2

}]

}

Iwillnowintroduceyoutothebasicsofthequerysyntax.Iwon’tcovereach

andeveryaspectofit.Itisfairlyrobustandmuchofitisbeyondthescopeof

this book and something you might not ever need to use. For example, I will

only briefly mention how to use the MongoDB aggregation features and its

syntax. For more information on that topic and other supported syntax

intricacies,refertoMongoDB’sdocumentation.

Syntaxoverview

Inalaterchapter,youwillbequeryingacollectionusingaMongoDBNPM

moduletowritecodeinJavaScriptthatrunsinNode.js.Thatmoduleprovides

functions such as find() or findOneAndDelete(). The first parameter of those

functionsisthequerycriteriathatspecifiesthematchingtotakeplaceacrossall

ofthedocumentsinacollection.

IfyouarefamiliarwithSQL,thisissimilartowhataWHEREclausedoes.

Hereisanexamplequeryusingthefind()functionwithagreaterthanoperatorin

the query criteria. This query below will return all of the documents in the

collectionwhoseagepropertycontainsavaluegreaterthan35.Thisisshowing

youhowyouwouldcallitincode,andthisalsohappenstobetheformatyou

useintheMongoshell.

db.collection.find({age:{$gt:35}});

For the function call above, there is the possibility that the query criteria

won’t match anything. This is ok, and no documents will be returned. On the

other hand, if there are a lot of documents returned, then you need to use the

Node.jsdrivercapabilitytofetchresultsinbatches.Youwillseehowtoactually

accesstheresultsofthefind()functionincode.

BesidesthequerycriteriaIjustshowedyou,therearealsocriteriayoucan

provide for what is called the projection criteria. The projection criteria

determine the properties that will be returned from each document. Here is

anotherexampleusing the samefind()function, butthistime withanoptional

secondparameterthatspecifiestheprojection:

db.collection.find({age:{$gt:35}},{name:1,age:1});

This query will find all documents in the collection whose age property

containsavalue greaterthan 35.Withthe projectionspecified, onlythename,

age and _id properties will be returned. _id is always returned, unless you

specifyotherwise.

Asmentioned,thesecondparameterin thisqueryisthe projectioncriteria.

This is where you list the properties you want to be returned. The number

followingthecolondeterminesifthepropertyisincluded,orifitisexcluded.I

willexplainmoreaboutthissoon.

Iwillkeepusingtheexampleofthebookstorecustomerdocument,butfor

now, just pretend they only have four properties each (_id, name, age, and

email).The followingdiagram of theprevious query showsthe two criteriain

thefunctioncallandhowtheydeterminetheoutput:

Figure14-Criteriaflow

Youcanseethatthequerycriteriadeterminewhatdocumentspassthroughto

the result set. The projection criteria select what properties you want for each

documentintheresultset.

Now I can go into the details of both the query and the projection criteria

operations. You can connect to your MongoDB hosted cluster through the

Compassapplication,addadatabase,addacollectionandsomedocuments,and

thentryoutsomequeriesonyourown.

Note:You cannotjust create a query and assumethat it will endup being

efficient.Theexecutiontimeofaquerycanvarygreatly.Toaddressperformance

issues, you either must create indexes that can speed up your queries or think

aboutamoreefficientwayofmodelingyourdata.Thetopicofindexcreationis

coveredlater.

3.1QueryCriteria

The query criteria are actually optional on a function such as the find()

function.Itis,however,somethingyouwillalmostalwaysbeusing.Ifyoucall

find() without any query criteria parameter, every single document in the

collectionwillbereturned.

The query criteria are tests that are applied to the collection to see what

documentsaretobeincludedaspartoftheresultset.Ifyoureallywantto,you

cannarrowdowntheresulttoreturnasingledocument.Forexample,youcan

querybythe_idpropertywithan equalitytest.The _idpropertyis unique,so

eachdocumentcanbeuniquelyidentifiedwithit.Hereisanexampleofcodeto

queryby_id.Theresultitreturnsisalsoshown:

//Query

db.collection.find({_id:{$eq:"77"}},{address.state:1});

//Results

{

"_id":"77",

"address":{

"state":"KS"

}

Youdon’tneed to writeany code totry this out, butcan use the Compass

applicationtotryoutqueries.Youcangotochapter6tolearnhowtocreateyour

PaaS hosted MongoDB cluster, database, and collection. You will also find

informationoninstallingthemongoshell.Youcanusethatknowledgetoimport

documentstouseasyouexperimentwithqueries.SeetheExampleDocuments

section,afewpagesprevious,forthedocumentstocreate.

Onceyouhaveyourtestdatabase,testcollection,withdocumentsimported,

you can launch Compass and go to your collection and in the Documents tab,

entersometestqueries.HereiswhattheCompassUIlookslikeifyouaretrying

outaquery:

Figure15-Compassapplicationqueryingcapability

Each query criteria can utilize one or more operators. The example above

usesthe$eqoperator.The real powerof the query is inthe use of thecriteria

operators.I’llnowgooverwhatthoseareandshowyousomeexamples.

Criteriaoperators

Youhave seen operators such as $gt and $eqused in the examples in this

chapter. Those operators stand for greater than and equal to. There are many

moreoperatorsthatyoucanuseinyourquerycriteriatofilterdocuments.The

followingoperatorsarecurrentlysupported:

Comparison

$eq

$gt

$gte

$in

$lt

$lte

$ne

$nin

Array

$all

$elemMatch

$size

Bitwise

$bitsAllClear

$bitsAllSet

$bitsAnyClear

$bitsAnySet

Element

$exists

$type

Evaluation

$expr

$jsonSchema

$mod

$regex

$text

$where

Geospatial

$geoIntersects

$geoWithin

$near

$nearSphere

Logical

$and

$or

$nor

$not

Eachoftheoperatorsusesitsownuniquesyntax.The$eqoperatorusesthe

followingsyntax:

{<name>:{$eq:<value>}}

Thenameiswhatyouwanttotestagainst.Itcanbeatop-levelproperty,orit

can be a property within the hierarchy of the JSON. The value is a string,

number or other value that matches the data type of the property. Here is an

examplethattestsasecond-levelproperty;referredtoasanembeddeddocument

fieldinMongoDBdocumentation:

{"address.state":{"$eq":"UT"}}

Youcanevenspecifyapropertyofanelementfoundinanarray.Ifyoulook

at the example document, you see that booksPurchasedis an array and idis a

propertyofeachoftheelementsofthatarray.Intheexamplebelow,thereturned

documentisthecompletedocument,asthequerycriteriaareonlyusedtofinda

documentmatchandnotspecifypropertiestoreturn.

{"booksPurchased.id":{"$eq":"1098"}}

Hereareafewoperatorsfromsomeofthecategoriestogiveyouanideaof

how they work. For more detailed information on each of the operators, see

MongoDB’sdocumentation.

Comparison

With the comparison operators, you need to first select the property name

you are interested in testing against. This is then followed by the comparison

operator and finally the value you want for that comparison test. The only

exceptiontothisiswiththe$inand$ninoperators,whichuseanarrayandnota

singlevalue.

IwillusethesameexampleIhaveshownyoupreviously,whichisdoinga

queryforasingledocumentbyits_idvalue:

{"_id":{"$eq":"77"}}

To just test the equality of a property, you can shorten the syntax to the

following:

{"_id":"77"}

Youcanusemorethanonecomparisonoperatoratatime,suchasyouwould

needtodototestrangesof values.Alloperatortestsneedtopasstheir testin

orderforagivendocumenttobeincludedintheresultsset.

Here is an example that queries for books that have between 100 and 200

pages:

//Query

{

"pages":{

"$gt":100,

"$lt":200

}

Logical

Thelogicaloperatorsletyoustringtogetherseveraltestsinarowtoperform

thedesiredlogicaltesting.Thesyntaxforthelogicaloperator$andis:

{$and:[{<expression1>},{<expression2>},...,{<expressionN>}]}

The logical operator syntax starts with a boolean operator such as $and.It

then contains an array of expressions that can be made up of individual

comparisonoperatorsthatwehaveseenpreviously.

Thefollowingquerylooksforbooksthatarelessthan200pagesinlength

andwhicharealsointheSoftwareEngineeringcategory.

//Query.

{"$and":[{"pages":{"$lt":200}},

{"category":{"$eq":"SoftwareEngineering"}}]}

//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}

{

"_id":"1098"

}

If you are only carrying out this one level of boolean operation, then you

don’treallyneedthe$andoperator.Instead,youcanjustlisttheconditionsone

afteranother.Hereisthesameexamplequerywithoutthe$andoperator:

//Query.

{"pages":{"$lt":200},"category":{"$eq":"SoftwareEngineering"}}

//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}

{

_id":"1098"

}

Here is a query that uses both the $and and the $or operators. I’ll use the

complete expanded text formatting of the query as it is easier to read. This

example queries for books that have less than 200 pages and are either in the

categoryofSoftwareEngineeringorScienceFiction.

//Query.

{

"$and":[

{

"pages":{

"$lt":200

}

{

"$or":[

{

"category":{

"$eq":"SoftwareEngineering"

}

{

"category":{

"$eq":"ScienceFiction"

}

]

}

]

}

//Results.Assumingyoualsohaveaprojectioncriteriaof{"_id":1}

{

"_id":"1098"

}

Ifyoufindyourqueryhasalotof$oroperationstomatchonmanydifferent

values for the same property, then you can use the $in operator. The value to

matchcanevenbearegularexpression.Thefollowingexampleshowshoweasy

itistousethistolistallthepossiblematches:

//QueryusingIN

{

"category":{

"$in":[

"SoftwareEngineering",

"ScienceFiction"

]

}

Element

Theelementoperators$existsand$typeareforselectionsbasedonwhether

apropertyexistsandifitisofacertaindatatype.Thefollowingexampleshows

thesyntaxfor$exists:

{name:{$exists:<boolean>}}

A typical use for $exists would be to use this operator inside another

operator. In this example, you want to make sure the document has the

publicationDateproperty.Itmaybethatabookhasapriceset,buthasnotbeen

publishedyet,sothepublicationDatepropertyisnotthereyet.

{"$and":[{"price":{"$lt":30}},{"publicationDate":{"$exists":true}}]}

Evaluation

Ifyoustruggletogetexactlywhatyouwantinyourquery,youmightfind

theevaluationoperatorsarejustwhatyouneed.Hereisanexamplethatshows

theuseof aregular expressionwiththe optionfora case-insensitivetest.This

examplewillfindallbooksthathaveatitlethatstartswiththeword“agile”no

matterthelettercasing:

{

"title":{

"$regex":"^agile",

"$options":"i"

}

You might have a document with large amounts of text that you want to

searchtoseeifspecificwordsorphrasesexist.Youcanusethe$textoperatorin

this case. To use this, you need to first create an index of type text on the

propertiesyouwanttouseiton.Forexample,youcouldcreatetheindexonthe

titlepropertyandthensearchforbooksthatcontaincertainwordsintheirtitle.

{"$text":{"$search":"MongoDB"}}

Forthoserareoccasionswhereyoujustcannotgetwhatyouwantwithany

oftheavailableoperators,youcanresorttowritingJavaScriptusingthe$where

operator.Hereisatestthatcheckstoseeifapersonhaspurchasedmorethana

single book. This requires JavaScript because the length property of the

booksPurchasedarrayisonlyaccessiblethroughtheAPIreturnedobject.

{"$where":"this.booksPurchased.length>1"}

YoucannotpresentlytrythisqueryoutfromtheCompassappifyourhosting

issetuptousetheAtlasfreetier.Thesameistrueforyourcode.

Objectandarrayproperties

So far, the examples shown have been testing properties that are single

values,suchasastringoranumericdatatype.Butwhatifyouhaveadocument

that has a property that is an array of strings? What if you have an object

property?Evenbetter,whatifyouhaveapropertythatisanarrayofobjects?

The address object property in the bookstore customer example document

endsupasanembeddeddocumentinMongoDBBSON.Youcandoasearchfor

an exact match on an embedded document and specify individual names to

matchasshowninapreviousexample.

Forarrays,youcandoanexactmatchonthefullcontentsofthearray,orjust

on specific values existing somewhere within the array. If the array holds

objects,youcansearchfortheelemententryandsub-propertyoffofthat.Here

isapreviousexamplequerythatwasdoingthis:

{"booksPurchased.id":{"$eq":"1098"}}

Justasyoucantestmultiplesinglepropertyvalues,youcanalsodothatfor

propertiesthatarearraysobjects.Let’ssayyouwantedtosearchforbookswith

bookreviewsbyJoewherehegaveafour-starrating.Hereanexampleofhow

thatwouldlook:

{

"bookReviews.reviewer":"JoeSchmoe",

"bookReviews.rating":4

}

Forasimplearrayofstrings,youcouldmatchforthatexactarray.Tosearch

fordocumentswhereonestringentryinanarrayexists,youcoulddoanequality

test.Hereisanexampledocumentwithapropertythatisanarrayofstrings:

{

"favoriteColors":["green","red","blue"]

}

Toincludethatdocument,herearethequerycriteriayoucoulduse:

{"favoriteColors":"green"}

ArraysearchesandprojectioncapabilitiesinMongoDBareverypowerful.If

you take the time, you can learn how to match on things like an element in a

specificindex,ordosomethinglikereturnthefirstnumericelementthatislarger

thansomevalue.Youwillhavetolearnthatyourself,asitistrickytoexplainall

the nuances. See the MongoDB documentation. For example, look at the

$elemMatchoperatordocumentation.

Datatypemismatchproblem

Equality comparisons can end up producing an undefined outcome if the

propertydatatypespecifieddoesnotmatchupwiththetestvaluedatatype.For

example,youcannottestforanumbervalueonapropertythatisastring.

TheteststatementsyntaxofMongoDBdoesnotworkthesameasitdoesin

theJavaScriptlanguage.Thefollowingquerywillnotworkbecauseofthedata

typemismatch:

//Theselectionwillnotwork,asthe_idpropertyisastring

//inthedocument,andyouarecomparingitwithanumber

{

"_id":{

"$eq":77

}

With the following JavaScript code sample, you can see that data type

coercionhappens.Abooleantestbetweenastringandanumberactuallyworks

inJavaScript.

//JavaScriptuses"=="forequalitytesting.

//Coercionrulesapplyandbothequalitytestsevaluatetotrue

varv="77";

v=="77";//trueresult

v==77;//trueresultascoercionhappens

3.2Projectioncriteria

Justbecauseyouhaveyour querycriteriareturningtheproperresult setof

documents does not mean that you are done. You may also want to set up

projection criteria to just return the properties that you really need. You have

seenthisdemonstratedalready,butnowyoucanlookatthisinmoredetail.

There may be cases where documents with all of their properties are what

youactuallywantreturned.Thismaybethecasewithaverysparsedocument,

makingitreasonabletoreturnthewholedocumenteverytime.Withlarger,more

complex documents, you can benefit from restricting the properties being

returned. Limiting what properties are returned saves on the amount of data

transferred.

Inclusionandexclusion

Youhavealreadyseenaprojectioncriteriainuse,soyoureallyknowmost

ofwhatyouneedtoknowalready.Justtoreview,ifyoudon’tprovideprojection

criteria, then the complete document is returned. If you do provide projection

criteria,thenyoucanspecifytheinclusionorexclusionofwhicheverproperties

youwouldlike.Exclusionmeanstoreturnallpropertiesexcepttheonesyoulist.

Theinclusionandexclusionsyntaxisasfollows:

True means to include and false means to exclude. You cannot mix both

inclusion and exclusion in the same projection criteria. The only exception to

this is if you are using inclusion criteria, you can also specify one single

exclusionifitistoexcludethe_idproperty.

Here are some examples of different projection criteria with a comment

addedtostatewhethertheyarevalidorinvalid:

{"address.state":1}//Valid

{"address.state":0}//Valid

{"name":1,"age":1}//Valid

{"name":1,"age":0}//Invalid

{"age":0}//Valid

{"name":1,"_id":0}//Valid

AsImentioned,thisisextremelyhandy.Let’ssayyouwanttocreatealistof

peoplewiththeiraddresses.Youcouldusethefollowingprojectioncriteria:

{"name":1,"address":1,"_id":0}

Thefollowingistheresultsetreturned:

{

"name":"JoeSchmoe",

"address":{

"street":"21MainStreet",

"city":"EmeraldCity",

"state":"KS",

"postalCode":"10021-3100"

}

{

"name":"JaneDoe",

"address":{

"street":"100SBridgerBlvd",

"city":"Paradise",

"state":"UT",

"postalCode":"84328"

}

Missingproperties

Since MongoDB can be schema-less, it is possible that any number of

documentsinacollectionthatyouarequeryingmightnotevencontainthegiven

property that you have specified in your selection criteria. For example, it is

possiblethatbooksPurchasedisamissingpropertyinsomeofyourdocuments,

by your own design. This is important to consider when you are constructing

yourquerycriteriaandprojectioncriteria.

The following example query will return two documents, but the second

documentwillnothavethebooksPurchasedproperty.Thisisbecausethesecond

customerhasnotboughtanybooksyet.

//Projectioncriteria

{"name":1,"booksPurchased":1}

//Results

{

"_id":"77",

"name":"JoeSchmoe",

"booksPurchased":[

{

"id":"1098",

"title":"AgileProjectManagementwithKanban"

{

"id":"1099",

"title":"TheMerriam-WebsterDictionary"

}

]

}

{

"_id":"78",

"name":"JaneDoe"

}

Ifyoureallywantthisseconddocumentleftoutcompletely,iftheproperty

does not exist, use a query selector to only get those with a non-null value as

shownhere:

//Querycriteria

{

"booksPurchased":{

"$ne":null

}

Ofcourse,thepropertycouldstillexistandjustbeazero-lengtharrayandit

wouldbereturnedinthatcase.

Arrays

There is a special operator named $slice that allows you to return just

specificportionsofarrayproperties.Examinethefollowingdocumentthathasa

propertycontaininganarrayofcolors:

{

"favoriteColors":["green","red","blue"]

}

Hereareafewexamplesoftheuseofdifferentoperatorslike$slice,$,and

$elemMatchtopulloutdifferentelementsfromthearraypropertyshownabove:

//Returnfirsttwoelements

{favoriteColors:{$slice:2}}

//Returnfirstelement

{favoriteColors.$:1}

//Returnfirstelementthatmatches

{favoriteColors:{$elemMatch:{$eq:"red"}}}

Formoreinformationontheseoperators,seeMongoDB’sdocumentation.

3.3QueryingPolymorphicDocumentsina

SingleCollection

In the section on data modeling, I mentioned that you may decide to

normalizesomeofyourdata.Youmightliketostoredataincompletelydifferent

document types in the same collection. To do this, your query must always

includeawaytopickoutjustthedocumentsofaparticulartypethatyouwantto

havereturned.

Using the bookstore example, if you had the customer and the book

documentsinthesamecollection,youcouldincludeatypepropertyineach.

Thefollowingisanabbreviatedexampleshowingthisapproach:

//Customerdocuments

{

"_id":"77",

"type":"CUSTOMER_TYPE",

"name":"JoeSchmoe",

...

}

{

"_id":"78",

"type":"CUSTOMER_TYPE",

"name":"JaneDoe",

...

}

//BookDocuments

{

"_id":"1098",

"type":"BOOK_TYPE",

"title":"AgileProjectManagementwithKanban",

...

}

{

"_id":"1099",

"type":"BOOK_TYPE",

"title":"TheMerriam-WebsterDictionary",

...

}

Ineveryquery,youwouldneedtoincludequerycriteriathatwerespecificto

thetypeofdocumentyouneeded.Hereisanexampleofthatquerycriteria:

//Querycriteriaforretrievingallbooks

{"type":"BOOK_TYPE"}

Chapter4:UpdatingDocuments

The previous chapter covered the topic of querying or the ‘R’ for Read

operationsintheCRUDacronym.Thischapterwillcoverthe‘U’fortheUpdate

operation.Iwillgiveanoverviewhereofhowthesyntaxworks.

When updating a document, you can certainly provide the complete

documentforuploading.Asanenhancement,MongoDBallowsyoutodothings

likespecifyingasinglepropertytobeupdated.Therearemanyupdateoperators

you can choose from and they can be combined in a single atomic update

submissionforagivendocument.

To use the update capability of MongoDB, you first need to provide the

querycriteriatoidentifythedocumentordocumentstobeupdated.Thatcriteria

usesthesamesyntaxasalreadycoveredforthequerycriteria.Whatisnewhere

is that the update criteria is added as another parameter. Here is the update

criteriasyntax:

{

<operator1>:{<name1>:<value1>,...},

<operator2>:{<name2>:<value2>,...},

...

}

Note: Create and Delete operations are being skipped. Creation is just

providingtheJSONdocumenttocreateandadeletionoperationusesthesame

querycriteriasyntaxthatareaddoes.AllMongoDBCRUDoperationswillbe

coveredintheservicelayerdiscussion.

4.1Updateoperators

Thefollowingarethecurrentlysupportedupdateoperationsusedonsingle

valueproperties:

$currentDate

$inc

$max

$min

$mul

$rename

$set

$setOnInsert

$unset

Herearesomeexamplestoillustratesomeoftheseupdateoperators.Iwill

only show examples that update a single property at a time. Youcan combine

multipleoperatorsondifferentpropertiesinoneupdatesubmission.Thesewill

allbecommittedatthesametimeandwillresultinanatomicoperationatthe

documentlevel.

Therearemanyoperatorsyoucanuse,andIwillgiveexamplesofafewof

them.Foreveryupdatecall,youneedasthefirstparameterthequerycriteriato

identifythedocument(s).Thesecondparameterspecifiesthepropertytoupdate.

Ifthequerycriteriaidentifymorethanonedocument,theupdatehappensonall

ofthosedocumentsidentified.

Note: The Compass app does not allow the use of the update syntax right

now, only query searches work. That is why I am showing the examples with

code.

$set

The$setoperatoristhestandardwaytoreplacethevalueofaproperty.The

followingexamplesetstherewardsPointspropertytoanewvalueforthequeried

person:

db.collection.update({_id:"77"},{$set:{rewardsPoints:1000}});

Ifthepropertydidnotexistinthedocument,itiscreated.Thisoperatorcan

beusedtodoacompletereplacementofanyvalue,evendoingareplacementof

acompletearrayoranembeddeddocument.Italsoworkstoreplaceaspecific

propertyofanembeddedobject.

$renamecan be used to give a property a new name. $unset will delete a

property.

$inc

The $inc operator is used to change the integer value of a property by a

specifiedamount.Youcanaddorsubtractfromanyvalue.Usingthebookstore

example, you could add rewards points to a customer. Here is an example of

whatthatwouldlooklike:

db.collection.update({_id:"77"},{$inc:{rewardsPoints:10}});

$minand$max

The$minand$maxoperatorsareusedtotestagivenvalueandonlyreplace

it if the value is less than or greater than the test value, depending on the

operator.Hereisanexample:

db.collection.update({_id:"77"},{$max:{rewardsPoints:2000}});

In this example, if the rewardsPoints property had a value of 1000 to start

with,ithasavalueof2000afterthisupdate.

4.2ArrayUpdateOperators

Thefollowingoperatorsareforusewitharraypropertiestoperformupdates:

$[]

$[<identifier>]

$addToSet

$pop

$pull

$pullAll

$push

The$pushoperatorisusedtoaddanotherelementtoanarrayproperty.Here

isanexamplethataddsanewbookreviewtoabookdocument:

db.collection.update({_id:"1098"},

{$push:{bookReviews:{

reviewer:"Skylar",

date:"20150923",

comments:"Itwasreallyprofound!",

rating:5

}}});

The $addToSet operator is similar to $push except it checks to see if an

identicalentryexistsalreadyandonlyaddsthenewelementifitisnotalready

presentinthearray.Youcanusetheadditionaloperatorof$eachtoaddmultiple

elementsatonce.The$sortoperatorcanbecombinedwiththe$pushand$each

operatortokeepthearraysorted.Combinethe$positionoperatorwith$pushto

specifythepointofinsertion.

The$ operator is used to specify that the update is to happen for only the

firstelementofanarraythatisfoundtomatchthequerycriteria.Thepartofthe

$setthat uses bookReviews.$.rating uses the .$to signifythat thereplacement

should take place on just the first element match that is found. This example

doesasearchforanydocumentthathasabookReviewselementthathasarating

of4andthenupdatesthefirstmatchedelement:

db.collection.update({_id:"1098",bookReviews.rating:4},

{$set:{"bookReviews.$.rating":5}}

)

$pop

The$popoperatorisusedtoremoveelementsfromanarrayproperty.This

exampleremovesthefirstbookreview:

db.collection.update({_id:"1098"},{$pop:{bookReviews:-1}});

Youcanuse$pulltoremoveallentriesfromanarraythatmatchwhatyou

specify.Youcan use$pullAll to specify more than one match forthe removal

criteria.

4.3Transactions

A single write operation might modify multiple documents if the selection

query matched more than one document. The modification of each individual

documentisatomic,butitisnotatomicacrossallofthemodifieddocuments.It

is possible to isolate a single write operation across multiple documents using

the$isolatedoperator,butonlyifthereisnoshardinginvolved.

Youmightalso have aneed to modify twodocuments simultaneously, and

evenmodifydifferentpropertiesoneach. Ifyouhavearequirement tochange

twodocumentssimultaneously,thenyouneedtoworkthisoutonyourown.For

example,ifyouwantedtotakerewardspointsfromonedocumentpropertyand

add them to a different document, this would require what is termed a multi-

documentatomictransaction.

This is a new capability in MongoDB 4.x. Previously, you would need to

build that capability yourself. Search “two-phase commit” as per MongoDB’s

documentation.

Youcanimaginehowimportantthiswouldbetogetrightinanapplication

that manages financial transactions across financial accounts. MongoDB now

supportsmulti-documentACIDtransactions.Youshouldonlyturnthisonifyou

reallyneedit.Thepointisthatadocument-baseddatabasemightnotneedthisin

mostcircumstances.Inthecode,youwoulduseanAPIcalltostartasessionthat

willsurroundseveraloperationsthatyouwanttobetransacted.Thenyouwould

eitherabortitorcommitit.Basically,somethinglikethis:

s=client.start_session()

s.start_transaction()

…codetodomultipleupdates,insertsetc.ondifferentdocuments.

--->onanerrorexceptionyouwouldcalls.abort_transaction()

--->onsuccessyouwouldcalls.commit_transaction()

s.end_session()

Chapter5:ManagingAvailabilityand

Performance

Inanidealworld,youcouldstoreaninfiniteamountofdata,accessitfrom

anywhereinnear-zerotimeandneverhaveanydatalossorcorruption.Realityis

that it takes a lot of work to approach these ideals. As you design your data

modelandsubsequentlytryitout,youneedtotuneyourDBMSforconsistency,

availability, and performance. You can now consider what mechanisms are at

yourdisposaltoapproachtheseideals.

It really takes a fair amount of time to fine-tune each aspect of the

managementof your MongoDB database. Many times, you will be faced with

tradeoffsthathavetobemade.Thischapterwilllookatsomeoftheaspectsthat

canbe“fine-tuned”forspecificaccessscenarios.

In a PaaS environment, some of this work should be less than has been

traditionallyrequiredinthepastwithaDBMS.MongoDBhascertainlydonea

greatjobofmakingsomethingsautomaticallyhappenthatusedtorequirealot

ofmanualconfiguration.

5.1Indexing

Imagine you had a problem finding personal belongings such as your car

keys, the TV remote, your favorite pair of socks, your wallet or purse, etc.

Perhaps when you try to head out the door, you find yourself frantically

searchingforyourcarkeyseverymorning.

One approach to solving this would be to keep a whiteboard right next to

yourfrontdoorthathadtwocolumns.Onecolumnwouldlisttheitemyoucared

about and the other column would list the known location of that item. The

whiteboardmightlooklikethis:

Carkeys

Left side pocket of the jacket hanging in the entryway

closet

TVremote Inthepileoftoysintheroomofyourtwo-year-oldtoddler

Purple

socks

Laundryroomfloorunderthepileoftowels

Wallet UnderthecouchcushionintheTVroom

Imaginethehugetimesavingsthiscouldprovide.Isawonestudythatstated

that,onaverage,apersonspendsawholeyearofaccumulatedtimelookingfor

lostitemsduringtheirlifetime.

A database index uses the same concept as the whiteboard look-up table,

with the goal of serving database records faster. A database index works by

creatingaseparatelookuplistthatallowsforfasterquerying.Thisalleviatesthe

needtosearchthroughalldocumentstofindtheone(s)youarelookingfor.For

example, let’s say you had a lastName property in every document of a

collection.Ifyoucreatedaquerythatwaslookingforaparticularpersonwith

thelastnameof“Smith”,howwouldaqueryfinditasquicklyaspossible?The

slowestwaytosearchwouldbetostartlookingatallthedocumentsonebyone,

untiladocumentwasfoundwith“Smith”inthelastNameproperty.Thattypeof

search has no choice but to search each and every document in an unsorted

storage system. In a huge collection, this would be a major performance

problem.

Indexingcanspeedupyoursearchbycreatingaseparatesortedlistoflast

namestosearchagainst.Eachentrywouldpointtothecorrespondingcomplete

document.A queryfor thelastname ofSmithquickly findsthose entrieswith

somethinglikeabinarysearch.

Asanexample,hereisarepresentationofrandomdocuments.Thereisone

rowperdocumentthatexistsinaMongoDBcollection.Thisisonlyanabstract

representation of how it is stored. To search for “Tuttle,” you would start a

sequential search from document to document until you found a match on the

lastName.Unfortunately,inthiscase,itwouldbethelastonefound.

Figure16-Customerdocumentsunsortedandwithnoindex

IfyouaddasecondlistthatcontainsthelastNameproperty.Alongwiththe

name,youcouldhavealinktothecustomerdocument.Thissecondlistcanbe

sorted, and your searching will be much faster. A quick binary search of the

index list will find the lastName to match and then use the link to get the

document.

Implementingagoodindexcanbeacriticalpartofyourworktomaximize

theefficiencyofyourqueries.Itiswellworthyourtimetomeasureandanalyze

the performance of your database and to fine-tune it with indexes. There are

reportsin thepaid Atlastier subscriptionsthat youcan bring upthat helpyou

lookatindexperformance.

Thefollowingexampleillustratestheconceptoflinkingthroughanindex.

Figure17–LastnameindexforCustomerDocumentquerying

YoucanusetheCompasstoolforcreatingallyourindexes.Thereareways

todothisprogrammatically,butforareasoftheapplicationthatonlyneedtobe

setuponce,IalwaysprefertodothisinCompass.

Note:IndexconfigurationcangetrathercomplexandIcanonlycoverthe

basiccommonscenariospertainingtothesampleapplication.Youwillhaveto

gototheonlinedocumentationtogetallthedetailsonwhatispossible.

Single-propertyindex

Thesimplestwaytolearnaboutindexesistolearnhowtosetupanindexon

a single property. This section goes over how this works for a single value or

object property. The next topic explains the subtleties of what happens if the

propertyisanarraydatatype.

Let’s go back to the example of the online bookstore. If you look at the

requirementsforyourquerying,youcanseethatyouneedtobeabletosearch

for customers by name. If you had hundreds of thousands of customers, it is

certainlygoingtoimprovetheperformanceofthisqueryifyoucreateanindex

onthenameproperty.

Thesyntaxusedtocreateanindexissimilartothesyntaxusedtosetupyour

othercriteria.Thisexampleshowshowtospecifyanindexonname:

{"name":1}

That is how simple it is. You can also add an index on a property of an

embeddeddocument.Forexample,whatifyouwantedtolookupallcustomers

thatresidedinacertainpostalcode?Tomakethisqueryrunfaster,youwould

wanttoplaceanindexonthesub-propertyasfollows:

{"address.postalCode":1}

Youcouldplaceanindexontheaddresspropertyasawhole,butthenyou

wouldhavetoputinacompleteaddressforthequerycriteria,includinghaving

thepropertiesinthesameorderforthematchtosucceed.

Note: The _id property that exists on every document is automatically

indexed,soyouneverneedtoaddanindexforthat.

Arraypropertyindex

Ifadocumentpropertyisanarray,yousetupanindexonitinthesameway

asforasinglevalueproperty.Therearejustafewrestrictions.Onerestrictionis

thatyoucannothavemorethanonearraypropertyindexatatime.Forexample,

ifyourdocumentshavemultiplearrayproperties,youcanonlysetupanindex

ononeofthearrayproperties.

Theindexwouldthenbeusedtomatchonanythingfoundinthearray.The

indexcanonlybeusedforindividualelementmatchingandnotformatchingon

thearrayasawhole.Asanexample,let’ssayyouhadthefollowingdocuments:

{"name":"Kiara","favoriteColors":["blue","yellow","cyan"}

{"name":"Tristan","favoriteColors":["green","red","blue"}

{"name":"Halli","favoriteColors":["juju","nana","mango"}

Youcansetupanindexsimilartothepreviousexample:

{"favoriteColors":1}

Nowyoucansetupaquerywithquerycriteriatomatchacolorthatmight

befoundinthearray.Ifyousearchfor“red,”thenthedocumentforTristanwill

be included in the results set. What is happening internally is that the index

contains all array element entries across all documents. There would be three

entriesintheindexforKiara,alphabeticallysortedbythecolorname.Eachof

thosethreewouldpointbacktotheonedocument.Similarly,therewillbethree

entriesforTristanandHalli.

You can also set up an index even if the array contains elements that are

objects,aswiththefollowingexample:

{"name":"Skylar","clothes":[

{"type":"schooldress","color":"tan","size":5},

{"type":"schoolpants","color":"tan","size":6},

{"type":"churchshoes","color":"white","size":3}]

{"name":"Korver","clothes":[

{"type":"pajamas","color":"green","size":3},

{"type":"tennisshoes","color":"brown","size":2},

{"type":"wintercoat","color":"blue","size":3}]

The following example will set up an index just for one property on the

objectinthearray:

{"clothes.color":1}

Multiple-propertyindex

Many times, you will have queries that specify more than one property to

matchon.Thisisusedwhenyouwanttonarrowdownyoursearchevenfurther.

Whatifyouwantedtorunbookstorespecialstoencouragepeopletouserewards

points.

Forexample,whatifyouwerelookingforallpeoplelessthan20yearsold

thathadnorewardspointssoyoucouldgivethemsomepointsasafreebonus

offertotrythemout.Thefollowingexamplecreatesanindexonbothageand

rewardsPointssoyoucansearchforcustomersthatway:

{"age":1,"rewardsPoints":1}

This is called a compound index. Be sure to understand how this really

works.Theunderlyingindexhassortedentriesforageandthen,hassub-sorted

entriesbyrewardsPoints.Thismeansthatyoucannotusethisindexforaquery

forjustrewardspoints.Findingallpeoplewithover1000pointswillnotbeable

to use the index. You can, however, use this index to query just by age. This

meansthattheorderofpropertieslistedfortheindexisimportant.

Youcouldcreatetwoseparateindexestobeabletoquerybybothageand

rewards points separately as well as combined in either order. MongoDB will

usesomething called an index intersectionfor you if it can.For example, you

couldcreatethefollowingtwoindexes:

{"age":1}

{"rewardsPoints":1}

Withthis configuration, you can still query by the combinationof age and

rewardspoints.Butyoucanalsoqueryjustbyage,orjustrewardspoints.You

canalsoreversethecombinationandquerybyrewardspointsandthenage,in

thatorder.Theonlydownsideisthateachindexyousetupmeansmorestorage

overhead.

Indexsortorder

Uptothispoint,Ihavealwaysbeenusingthenumericvalueof1inallofthe

indexexamples.WhatthatisdoingisinstructingMongoDBtocreatetheindex

in ascending sort order. You can alternatively specify -1 and get a descending

sortorder.Forexample,youcouldcreatethefollowingindex:

{"age":-1}

Forsingle-propertyindexes,thisdoesnotmatter.Itonlymattersifyouwant

toreturnmultipledocumentsofmorethanonepropertythroughacorresponding

multi-propertyindexandyouwanttheresultreturnedinaspecificsortorder.For

example,queryingforallpeoplewiththelastnameof“Smith”andreturningthe

documents in descending order by age. Otherwise, you can certainly add a

commandtosorttheresultthatisreturned.

With the API usage, there is a sort() function that can follow the find()

functionthatcanprocessthesortorderforyou,butitwillbedoneoutsidethe

indexandbeslower.

Othertypesofindexes

Theindexexamplesusedsofarwouldbeusedforexactvaluematchesand

rangetypequeries.Justsoyouareaware,thereareafewothertypesofindexes

thatyoucaninvestigateandutilizethatmaysuityourneeds.Someoftheseother

indextypessupportedbyMongoDBaregeospatial,hashedandtextindexes.

Forexample,let’ssayyouwerekeepingadatabaseofrestaurantsalongwith

their menus, reviews, and location coordinates. With a geospatial index, you

could then perform queries to take a person’s current location and find all

restaurantswithinacertainradiusofthem.

Ifyouhavelargeamountsoftextinaproperty,youcanusewhatiscalleda

textindex.Forexample,ifyouwerestoringnewsstoriesandyouwantedtohave

anindexonthestorycontent,youcouldfindatextindexveryperformant.

Thereisanindexcalledahashindex.Itcanbeusedonpropertiesthatholda

single value (i.e. string, number) or that have an embedded document, but it

cannot be used with array properties. The hashed index is populated with the

hashedvalues.Ahashindexbyitselfdoesnotworkforrange-basedqueries,so

youmightwantanotherseparatesingle-propertyindexforthat.

Forembeddeddocuments,theindexgivesahashofthecompleteembedded

document. This then would potentially be a faster lookup as the search is just

comparinga hashvalue to findthe document.Ahash indexmight make more

sense when combined with sharding. This will be covered in an upcoming

chapter.

Indexcreationoptions

Oneoftheoptionsyoucanusewiththecreationofanindexistospecifythat

the values for a given property are unique. For example, in the bookstore

example,eachcustomerhasanemailproperty.Youcouldmaketheindexrequire

that all emails must be unique across all documents. Here is an example that

createsanindexwiththeuniqueoption:

{"email":1},{unique:true}

You should create an index like this before any data is populated. Index

creationwouldfailifmultipledocumentsexistthathavethesamevaluefortheir

emailproperty.Onceanindexiscreatedwiththeuniqueoption,anydocument

creation will fail unless it has a unique value on the indexed field with this

optionset.

Thesparseoption can be used to create an index that only has entries for

documentsthat contain that property. In the following example, if a document

didnothavetheemailproperty,thenitwouldnotbeincludedintheindex.Any

subsequent query for values to match on using the email field would ignore

thosedocuments,butthatisprobablywhatyouwanted.Ifanindexexistsfora

property, then MongoDB will use it, so that is why the rest of the documents

wouldnotbesearchedthataremissingthatproperty.

{"email":1},{sparse:true}

Indexcreationoptions

That about wraps up the topic of indexes. As I stated at the start of this

chapter,therearenoperfectsolutionsintheworldofdatabases.Thisistruewith

indexing. What you need to do is to completely understand what your query

needsarefirstinordertounderstandhowbesttocreateyourindexes.

For example, you might even consider not having any index under certain

circumstances. If you had a database with 95% of the operations being writes

and5%werereads,youmightnotwanttocreateanindex.Thisisbecausean

indexwillslowdownyourwriteoperations.Thereare,however,configuration

settingsyoucanstillmaketospeedupwrites.

Ifyouhavetheoppositesituation,andreadperformanceneedstobefast,and

writesareasmallpercentageoftheload,thendefinitelycreateindexes.Youcan

always measure your performance before and after to make sure that your

indexingdecisionsarevalid.

Note: Once your database is up and running and you are running code

queries against it, you can get a diagnostic report on an individual database.

Thiswilltellyouhowwellyourindexisperforming.Thisisdonebyeitherusing

theexplainoptionortheexplain() method, dependingon the APIcall you are

making.

5.2AvailabilitythroughReplication

Asinglepointoffailureisnevergood,nomatterwhatserviceyouareusing.

Forexample,inthedaysofthetelegraph,theremighthaveonlybeenasingle

telegraphlineconnectingtwocities.Thatconfigurationcreatesaservicethathas

a single point of failure. Cut the single line and communication is severed.

Havingtwotelegraphlineswouldgiveyouredundancy.Itwouldalsogiveyou

greaterthroughputifyouputbothintousageatthesametime.

Thesameredundancyisnecessarywithdatastoredonharddrives.Perhaps

allyourfamilyphotosareonasingleharddrive.Whatifthatharddrivefails?

Doingbackupstomakecopiesisnecessary.Anyapplicationdatamustneverbe

at risk of being lost or being unreachable. Therefore, some form of data

redundancyisnecessarywithMongoDB.

MongoDBreplicaset

If you have a single machine for your MongoDB database, you could

configure a MongoDB single-node. Then when that goes down, you cannot

accessyourdatabaseuntilitcomesupagain.Thiswouldbefineforoccasional

usagescenariosorfordevelopmentexperimentation.

MongoDBhastheabilitytoconfigurewhatiscalledareplicaset.Thisgives

youmultiple parallel,redundant copies ofall data. Thisensures that yourdata

willbe safe and available.This is what youalways get by defaultthrough the

AtlasportalifyouutilizethatPaaS.

Inareplicaset,youhavemultipleduplicateddatabases.Onlyonemachineis

designatedastheprimaryatanygiventime.Iftheprimarydatabaseservergoes

offline, a secondary server will take over. The following diagram gives you a

generalideaofwhatthislookslike:

Figure18-MongoDBreplicaset

Theconfigurationwilllookslightlydifferentbasedonwhichplanyouselect

from Atlas. The basic idea is that the primary database server receives and

fulfillsallreadandwriterequests.Allthewhile,thesecondarydatabaseservers

are kept up to date with all changes. Each database server can be on its own

dedicatedAWSEC2virtualmachineindifferentavailabilityzones.

Each server is constantly being checked with a heartbeat signal. If the

primary database goes down, the two secondary servers and the arbiter would

detectthat,andoneofthesecondaryserverswouldbeswitchedovertobecome

theprimarydatabaseserver.Thearbiterisreallyjusttheretobreakanytievotes

ifneeded.

Areplicasetallowsforfasterreadingofdatabecausemultiplecopiesexist

anddata can be fetched in parallel from each replicacopy, if that is what you

want.Youmust designatereads tobe fulfilledby the secondaryservers if you

determinethatisjustified.Justbeawarethatyoucouldgetstaledatathathasnot

yetbeenupdatedbyareplicationprocess.Thisisgreat,becauseyoukeepadding

secondarymachinesandcanachievebetterreadperformancenow.

Note: With a PaaS solution, you generally do not have control over the

replica set configuration unless you work with the provider to get something

customized. There are pre-determined configurations you select when you

purchaseaplan.NothingispreventingyoufromimplementinganIaaSsolution

andsetting up your own virtual machines and replica set configurationif that

wouldworkbetterforyou.

Secondaryconsistency

Thereisacomplicationtobeawareofinhavingreplicationsavailable.Any

writetotheprimarystoragecollectionmusteventuallymakeittoallthecopies.

Therefore,youmustmakeachoiceastohowthatreplicationisaccomplished.

MongoDBhasasettingcalled“writeconcern”thatallowsyoutospecifyif

youwantamajorityofreplicastoreportthatthewritehastakenplacebeforeit

isacknowledgedorfailed.Youdon’thavetorequirethis.Ifyoudon’t,thenall

writeseventuallymaketheirwayasynchronouslytoallreplicadatabaseservers.

5.3Sharding

The replication previously discussed stores the same data on multiple

machines to provide emergency backup to ensure availability. Sharding also

spreadsdataoutacrossmachines. Withsharding,agivendocument appearsin

only one replica set of a sharded cluster, but would be in the primary and

secondarymachinesofthatshard.

Thepurposeofshardingistoallowyoutogrowtheamountofdatayoucan

store and also increase the performance of operations. Both concepts of

replication and sharding can be applied at the same time in an architecture.

Sharding is just the increasing of the number of replica sets that you have as

individualunits.

Shardingspreadsthedata acrossmultiplereplica sets.The multiplereplica

setsinashardedclusteractasiftheywereonesinglecollection.Thesharding

technologyknowswheretogoforanygivenreadorupdatetomakeiteasyfor

youtouse.

Sharding helps when you have large datasets and are wanting to maintain

high throughput. For example, you might have a lot of data constantly being

accessed.ThiscanbecomeabottleneckwithasingleSSD.Ifyoudistributethe

load across multiple SSDs, then the CRUD operations would not conflict as

much.

MongoDBcanbesetuptotakecareofeverythingforyou.Youcanselecta

planfromAtlasthathasitallsetupforyou.

Atlashasvariousplansyoucanchoosefrom,dependingonhowmuchyou

are budgeting to spend. With the free-tier plan, there is a hard limit with one

singleSSDblockstorageforyourdatabase,soyoucanonlygouptoacertain

sizeandthenyoucan’tgrowbeyondthat.Atlascurrentlyonlyoffersreplicaset

and sharded plans (multiple replica sets), and you can’t just have a single

MongoDBserviceonitsown.

Clusterplansgouptoacertainamountofstorage.However,workingwith

mLaborAtlassupportpeople,youcankeepincreasingthehorizontalscalingof

theshardingby addingmore storage.Additionalreplica setscan, intheory,be

addedtoaccommodateyourlargestdatastorageneeds.

Reasonsforsharding

Theconceptofdatasharding(alsocalledpartitioning)wasinventedtohelp

approachtheidealofbeingabletostorean“infinite”amountofdataandretrieve

any part of it in a minimal amount of time. Let’s dig a little deeper into the

scenarios that will cause you to implement a strategy for sharding. Here are

somereasonstoimplementdatasharding:

Runningoutofroom:Withalimittostorageforasingledatabase

SSD,youmightsimplyoutgrowthatcapacity.

Machine performance: There are utilization limits for RAM,

CPUandSSDaccessthatmightbereached.

Note: With an Atlas PaaS databases, you have the ability to choose a

configurationwithshardingalreadyconfiguredforyou.InthecaseofmLaband

Atlas,you can picka preconfigured machinearchitectureand then setup how

yourshardingwillact.IfyouwanttogotheIaaSroute,thenyoumustconfigure

thisyourself.

Howshardingworks

Here is how sharding works. Imagine that you start out with a single

MongoDBdatabaseserverandonthatserver,youhaveasinglecollection.Each

document you create could have a property that has a random capital letter

chosenfromAthroughZ.Youmightalsosetupanindexontheletterproperty.

AJSONdocumentyoumightwanttoinsertcouldlookasfollows:

{

"letter":"G"

}

Atthispoint,nomatterwhattheletterpropertyvalueis,alldocumentswill

be created in the same database collection. The box below represents a single

replicaset(primaryandtwosecondarymachines).Thisexampleshowswhatthis

wouldlooklike:

Figure19-Collectioninasinglereplicaset

Then, at some point, you realize that you need to add a whole lot more

documentsand want to achieve ahigher level of throughput on yourread and

writeaccess.Theabovesingle-nodeconfigurationcanthenbemadeintowhatis

calledamulti-nodeshardedcluster.

MongoDB will start balancing documents between the available shards

(replicasets)intheclustertocreateamoreevenlydistributedstorage.Itactually

does this in chunks. With additions and deletions of documents happening,

MongoDB keeps it all balanced based on the sharding key. You can choose

eitherahashorarangestrategyforyousharding.

Yourdocumentsendupbeingdistributedoverthethreeshardsinthecluster.

See the following figure for a visualization of the distribution using a range

shardingstrategy:

Figure20-Collectiondistributedoverthreeshards

Asitturnsout,eachshardisareplicaset.Whenadatabaserequestcomes

intothecluster,MongoDBdoesalltheworktoroutetherequesttotheproper

replica set shard. Your code is shielded from the fact that this is going on. A

singlelogicalcollectiondoesalltheworkforyoutocoordinateacrosstheactual

shardsthathavetherealcollections.

Iwon’tgointothearchitecturaldiagramshowingthecomponentstosetthis

up, but you can look it up online if you really want to implement an IaaS

configurationonyourowninsteadofusingthePaaSsolution.Whenyouusethe

AtlasPaaSsolution,youwouldmostlikelyenlistasupportengineertohelpyou

ifyouwantedtocustomizeyourshardedcluster.

Note:Thereisawaytotakeshardsoutofyourcluster.Thereisamechanism

to let MongoDB know that this is your intention. Once you do so, MongoDB

beginsmigratingdataoffofthesoontobedecommissionedshard.Oncethatis

done,thatshardcanbefreedup.

Shardingkey

Yourshardmustbesetupwithwhatiscalledashardingkey.Ashardingkey

issimilartohowanindexissetup.Withanindex,youspecifyapropertythat

youwanttouseforaspeedylookup,usingsomedeterminedalgorithm,suchasa

rangeorahashsearch.

Lookatthepreviousfigure,andyouwillseethreeshards.Theshardingkey,

inthiscase,isthepropertythatMongoDBwillusetodeterminewhatshardeach

documentexistsin.Forthisexample,itwouldhavebeentheletterproperty.

Eachdocumentcanonlyexistinonesingleshard.Ashardkeyisthusused

asasortingproperty.IfIcreatedadocumentwiththeletterpropertysetto‘M’,

it could be stored in the middle shard because of the range strategy setting it

there.

There is a fair amount of thought needed to select the proper sharding

strategyandselectapropertytokeyoffof.Justrememberthatyoumustknow

whatyourqueriesaregoingtolooklike.Don’tforgetthatyoumightevenhave

queries that cross shards, like those using range criteria. Imagine if you want

documentsfromthepriorexamplethathadalettergreaterthanDandlessthan

L?Theshardservicewouldactuallyknowitneedstosendthequerytoboththe

firstandsecondshardsandthenyourcodewouldprocessalloftheresultsetfor

whatyouwant.

Indexesstillexistoneachshard.Thequerylookupwouldfirstgotoashard

and then the shard replica set would use any applicable indexes to find the

document(s). You can have multiple indexes, but only one sharding key. The

shardingkeymustbethesameasoneoftheindexes.Inourexample,therewas

anindexfortheletterproperty,andthatwasalsousedfortheshardingkey.

Ifyouhaddocumentsrepresentingcustomers,youcouldlookpeopleupby

theirlastname.Youcouldthenuseahashedshardingkey.Thatway,queriescan

narrow the location to one single shard and then quickly retrieve documents

fromthat shard using the index. A hash shard key is nice because it can most

likely give you a more uniform distribution of documents across shards for a

fairly even retrieval cost. This is great for locating documents with a specific

querythatcanzeroinonthedocument.

Rangequeriesmightnotbeaseffectivewithsharding.Ifyoudoknowyou

haveagooddistributionofrangevalues,thenperhapsarangestrategywouldbe

best. Range sharding is efficient if you have queries where reads target

documentswithinacontiguousrangeofvalues.

You also must consider what your queries will look like and what your

document composition will look like. For example, you will certainly have a

performanceproblemwithhashstrategyshardingifyoutryanddoarangetype

ofquerythatcausesalltheshardstobesearched.

Ifyourquerydoesnotactuallyutilizetheshardkeyproperty,thentheservice

hasnochoicebuttosendthequerytoallshardsintheclusterandthencollectall

oftheresults.Butatleast,youshouldhaveconsideredwhattheindexshouldbe

formakingthateffective.

Likeanindex,ashardkeycanconsistofmultiplepropertynames.Youcould

thus use a compound key such as last name, first name, and city. Shard keys

cannotbecreatedforapropertythatisanarray.

Youmight have been thinking that if you kept adding documents with the

sameorsimilarkeyvaluethattheycouldallgotothesameshardandthenthe

diskforthatwouldrunoutofmemory.Thisisactuallynotthecase.Thereisa

process going on in the background, regardless of the sharding type (range or

hash) that moves data around between shards in chunks. This is called the

Balancer. You don’t actually set the shard boundaries yourself. MongoDB

figuresthatoutforyou.Figure20wasjustafictitiousillustrationofapossible

balancingthatcouldhappen.

Aswasmentioned,youareultimatelylimitedbythediskspaceavailableand

thatis whyyou wouldkeep adding shardsas youreach thelimit of document

storage. As a new shard is added, the Balancer does the work to spread

documentsevenlyoutacrossallavailableshards.

Note:Letmemakesureyouhavealltheterminologydown.TheAtlasportal

letsyoucreateacluster.Thisisthesetofmachinesthatholdyourdatabases.In

theirterminology,aclustercanbeeitherareplicaset(primaryandsecondary

machines), or a sharded cluster (multiple replica sets that are each called a

shard)

Chapter6:NewsWatcherApp

Development

This chapter takes some of the concepts that you have learned and applies

themtoaprojectusingtheAtlasPaaSoffering,hostedinAWS,tocreateadata

layer.Youwill learnhow toget thedata layerup andrunning and learnsome

bestpracticesalongtheway.

In this chapter, you will go to the Atlas portal and create the MongoDB

database and collection resources for the NewsWatcher sample application. To

getstartedyoumustfirsthaveanactiveAtlasaccount.

Note: MongoDB is an open-source project and you could download it for

freeandrunitonanymachineyoulike.Thisisnottheapproachtakeninthis

book. You can certainly investigate that option if it better meets your needs.

ThereareotherMongoDBPaaShostingoptionsouttherebesidesAtlas,sodo

yourresearch.

Youwillbesettingupthefollowingresources:

Figure21-NewsWatcherMongoDBresources

Theonlydocumentyouwilladdtothecollectionrightnowistheoneyou

shouldaddmanually.ItisrequiredforthefunctionalityoftheNewsWatcherapp.

Youshouldalsomanuallyaddafewotherdocumentsjustfortestingpurposesto

tryoutafewqueries.Later,youwillseehowdocumentswillbeaddedthrough

JavaScriptcodeinyourNode.jsprocess.

6.1CreatetheDatabaseandCollection

Thefirsttaskwillbetocreatethedatabase.Forthesampleapplication,you

canselecttheoptionthatwillgiveyoufreehosting.Thiswillbefineforyour

development and testing purposes until such time that you need to scale for

greaterstorageandperformance.

You can also study the other configuration offerings available through

MongoDB Inc. You can even try them out for a day or two, as you are only

charged for the time you have them available, and you can easily delete them

whenyounolongerwantthecharge.

Itiscertainlyworththecosttotryoutsomeoftheotherconfigurationsthat

allowforothercapabilitiessuchassharding.Youmightwanttotakesometime

to look through the plans and pricing pages on the MongoDB Atlas site to

familiarizeyourselfwithwhatispossible.Forexample,theamountofRAMand

storageand IOPs changes per the differentplans. Thus, it is veryimportant to

understandyourusageneedstobeabletoselectaplan.

When you sign up, you can select the hosting provider such as AWS, the

hosting location, and the specifics about the configuration that determine the

charge. To create a database cluster for the NewsWatcher app with the Atlas

portal for free hosting, you can go to the MongoDB Inc. website

(https://www.mongodb.com/). It is fairly self-explanatory from there. First,

createaprojectinyouraccount.ThenlookfortheBuildaNewClusterbutton.

Atsomepoint,youwillneedtoprovideausernameandpassword.

Aftercreatingyour databasecluster everythingwill bedeployed andready

foryourusage.Onceitisreadyyouneedtosetupaccessonyourmachinefor

theCompassdesktoptoolortheMongoshell.

1. ClicktheConnectbutton.

2. ClickADDCURRENTIPADDRESSandgiveitaname.Ifyour

IPaddresswillbeconstantlychanging,selecttoallowallIPaddresses.

You will still be secure, as access must be authenticated and

authorized.

3. DownloadtheMongoDBCompasstoolfoundintheAtlasportalat

thebottomofthepage,byclickingDownloadCompass.Thiswillbe

necessarylaterforyourinteractionswiththedatabaseandcollection.

ThefigurebelowshowsyouwhattheAtlasportallookslikeatthetimeyou

useittocreateadatabasecluster.

Figure22-CreateClusterpage,MongoDBInc.portal

Inafewmoments,youwillbeallsetupandwillbereadytostartusingyour

freeMongoDBdatabasefromMongoDBInc.,hostedonAWSEC2machines.If

youlookintheMongoDBAtlasportal,youwillseeyourclusternowshowsup.

SetupaconnectionwithCompass

WiththeCompasstoolinstalled,youwillneedtomakeaninitialconnection

toyourMongoDBhosteddatabase.Lookbackattheinformationyousawinthe

Atlas portal when you clicked CONNECT for your cluster. In there you will

find the hostname of your cluster. You can configure Compass to be able to

makeaconnectionasfollows:

1. In the Atlas portal, click the CONNECT button again and then

clicktheareathatsaysConnectwithMongoDBCompass.

2. ClicktheCopybutton.

3. OpentheCompasstool.

4. Clicktocreateanewconnection.Everythingshouldauto-populate,

andyoushouldbegoodtogobyclickingCONNECT.Ifnot,follow

thenextinstructions.

5. PasteintheHostname.

6. Theportshouldalreadybesetto27017.

7. SetAuthenticationtoUsername/Password.

8. Entertheadminusernameandpassword.

9. LeaveSSLtoUseSystemCA/AtlasDeployment.

10. LeaveSSHTunneltoOff.

11. EntersomethingtorememberthisconnectionbyfortheFavorite

Name.

12. ClickCONNECT.

Figure23-CompassconnectionUI

AddadatabaseandcollectionthroughCompass

You need to create a database with a collection inside of your cluster as

follows:

1. OpentheMongoDBCompassappandconnect.

2. ClickCREATEDATABASE(hoveroverthe“+”plussignatthe

bottom of the left pane) and type in the Database Name. I entered

“newswatcherdb”. Enter a name for Collection Name. I entered

“newswatcher”.

3. ClickCREATEDATABASEintheform.

Note:IfCompasswillnotallowyoutoperformedits,itisprobablybecause

youareconnectedtoasecondary,andnottheprimarymachineinthecluster.

You can now create the one required document that must be manually

created.Youwillseelaterhowthisdocumentfitsintoyourdatamodel.Tocreate

thedocument,dothefollowing:

1. Clickonthenewswatcherdbdatabase.

2. Onthenewswatchercollection,clickINSERTDOCUMENT.

3. Typeinthedocumentcontentasshownbelow.Makesuretoselect

adatatypeofArrayforthenewsStoryproperty.ClickINSERT:

{

"_id":"MASTER_STORIES_DO_NOT_DELETE",

"newsStories":[],

"homeNewsStories":[]

}

Figure24-Createadocument

Toaddanewproperty/field,youcanhoveroveranumberontheleftandit

willchangetoaplussignandthenyoucanaddone.Youalsoneedtochangethe

typewiththedropdownyouwillfindthere.AfterclickingINSERT,youwill

seethedocumentonthecollectionpage.Isn’tPaaSwonderful?Thereisnosetup

ormaintenanceorworryingaboutifyouhavethelatestversionofMongoDB.

Installofthemongoshell

The Compass tool does not now allow for easy creation or import of

documents,becauseyou havetocreate themaproperty atatime. Toimporta

largerdocumenteasily,youneedtoinstallthemongoshell.IntheAtlasportal

UI,youwillfindalinktodownloadthemongoshell.Youwillwanttoselecta

custominstallwiththeselectionslookingasfollows(seefigure25).Otherwise,

youwillgetthedatabaseinstalledlocallyalongwiththemongoshell.Youdon’t

actually need the mongo shell to go forward developing the NewsWatcher

application,sodon’tinstallitifyouarehesitantaboutit.

OntheUIpageofAtlasthatshowstheconnectioninformation,youwillfind

astringtocopythatgivesyoutheinformationnecessarytomakeaconnection.

Givethefullpathtothemongoexecutableandaddtheconnectionstringtothat.

Itlooksasfollowstorunthemongoshell:

"C:\Program Files\MongoDB\Server\3.4\bin\mongo.exe" "mongodb://cluster0-shard-52-78-

k6yhs.mongodb.net:27017, cluster0-shard-52-78-k6yhs.mongodb.net:27017, cluster0-shard-52-78-

k6yhs.mongodb.net:27017/test?replicaSet=Cluster0-shard-0" --authenticationDatabase admin --ssl --

usernamemax--passwordblah

Figure25–Installthemongoshell

6.2DataModelDocumentDesign

It is time to diagram out the structure and relationships of the document

typesthatyouwillneedfortheNewsWatcherapplication.Thisisdefinitelyan

iterative process where refinements are made over and over until it is correct.

Evenafter you have implemented a data model, you may find that it does not

give you the performance you expected, and you might end up altering the

design.

Think again about what the requirements are for the NewsWatcher

application and you can understand what is needed. NewsWatcher will have

users that log in. Thus, you have identified that there is a need for a user

document.

There is also a single document that holds the master list of news stories.

Therewillbesomecodethatisruneveryfewhourstocollectnewsstoriesand

storetheminthatdocument.

Athirddocumenttypeisforthenewsstoriesthatusersshareandcomment

on. There would be multiple User and SharedStory documents, but only one

MajorStoriesdocument.Thismodeliscompletelydenormalized,sothereareno

keystolinkanydocumentstogetherandtherewillnotbeaneedforanytypeof

joinoperations.Thefollowingdiagramshowssomeoftheneededdocuments:

Figure26-NewsWatcherdocuments

Let’s look at what the User document contains. In there, you will want to

includeanemailaddressforeachuser.Thiswilluniquelyidentifyyourusersand

allowsthemtosignin.Usersmustalsoenterapassword.Youcansafelystorea

hashedvalueofthepassword(youshouldneverstoreapasswordinplaintext).

Thenyoucanletuserspickadisplaynamethatotheruserswillseewhenauser

commentsonasharedstory.Youshouldneverrevealtheiremailtoanyoneelse.

Next, there should be certain global values that can be used for user

preference settings. You can put that in a sub-hierarchy called settings. For

example,you might want to give users the option of not using any cell phone

dataandrestricttheapptousingWi-Fionly.

Youcanassumethattherewillbesomeusersthatwouldlikeanalertfeature

for when news stories come in to be immediately notified. Youcould create a

Booleanvalueforthat.

ThecompellingfeatureofNewsWatcheristheabilitytohavetheappscan

forthenewsausercaresabout.NewsWatcherusersarenotthetypethatwants

to go to some general overall curated news page, but are interested in

customizing their own specific filtered view of their news. This is done by

filteringnewsstorieswithkeywords.

Userscan setup asmany filtersas theylike, soyou canconclude thatthe

designrequiresanarrayoffilters.Eachfilterwillneedtocontainatitleforthe

filter,keywords,timeofthelastnewsscanandalistofstoriesandtheirtimeof

capture.Thelistofstoriesforafilterispopulatedbyscanningthemasterstory

documenttoseeifthereareanymatcheswiththekeywords.

NewsWatcherhastheabilitytosaveoffinterestingstoriespereachuserso

thatthey appearseparately.This is what the savedStories property is used for.

Wewon’tactuallyimplementthatatthispoint.

The other properties shown in the user document are for other features as

outlinedintherequirements.Thoseotherfeatureswon’tbeimplementedthough.

ThiswillgiveyouagoodstartatanMVP(MinimumViableProduct)togo

outwith.Ifyoudoabitofadvancedthinking,youcanmodelallofthisinyour

diagramandjustnotimplementeverythingyet.Youcanfeelconfidentthatyour

datamodelcanaccommodateyourfutureneeds.

Enteringsometestdata

Atthispoint,youcanopenthepageforthenewswatchercollectionandadd

adocument to the collectionfor testing thingsout. Youcan of course onlydo

this if you had done the work to install the Mongo shell. You can insert

documentswithCompass,itwilljusttakeyoulonger.

1. Startthemongoshellasshowninsection6.1.

2. Attheshellprompt,youcantype“usenewswatcherdb”todirect

commandstothatdatabase.

3. Runthefollowingcommandtoinsertadocument:

db.newswatcher.insertOne(

{

"type":"USER_TYPE",

"displayName":"Bushman",

"email":"nb@hotmail.com",

"passwordHash":"XXXX",

"date":1449027434557,

"settings":{

"requireWIFI":true,

"enableAlerts":false

"savedStories":[],

"filters":[

{

"name":"TechnologyCompanies",

"keyWords":[

"Apple",

"Microsoft",

"IBM",

"Amazon",

"Google",

"Intel"

"enableAlert":false,

"alertFrequency":0,

"enableAutoDelete":false,

"deleteTime":0,

"timeOfLastScan":0,

"newsStories":[]

}

]

})

4. Run“db.newswatcher.find()”attheprompttoseeanydocuments

inthecollection.

5. Run“exit”toquitthemongoshell.

You will see the document added. It has an automatically assigned _id

created. You can launch the Compass application and view your created

documentsanddeleteoreditthemasnecessary.

For now, you can go ahead and experiment by creating a few more User

documentsinthissamecollection.Later,documentswillonlybeaddedthrough

code. At this point, all you are interested in, is being able to test out some

queriesbeforedevelopingthenextlayeroftheapplication.Youcangetafeelfor

howtheportalUIisusedandlearnabouthowqueriesareconstructedbeforeyou

putthoseintocode.

IfyoureadthroughtheMongoDBdocumentationyouwillfindthatthereare

ways to do bulk importing or exporting of documents. For example, there are

toolslike mongoimport andmongoexport that youcan run from thecommand

line.Forexample,anexportmightlookasfollows:

mongoexport -d test -c records -q '{ date: { $lte: new ISODate("2017-09-01") } }' --out

exportdir/myRecords.json

6.3TryingOutSomeQueries

Youmighthaveenteredafewdocumentsbyhandinacollection.Youcan

nowtryoutsomequeriesagainstthatdatathroughtheCompassapplicationor

theMongoshell.Atthispoint,youjustwanttogetafeelforwhatthetoollooks

likeandtobereadytolearnabouthowqueriesareconstructedbeforeyouput

thoseintotheservicelayercode.

You will use the sameUI shown in earlier chapters to run queries against

yourMongoDBcollection.Thatiswhereyouutilizethecriteriasyntaxtoquery

andspecifywhatyoudesiretoseeintheoutput.

Trysomequerieslikethefollowing:

{"type":"USER_TYPE"}

{"type":"USER_TYPE","email":"nb@hotmail.com"}

Now set the PROJECT in the options area of Compass to be

{"displayName":1}andtrythequeryagain.

Ifyourquerysyntaxisincorrect,youwillbenotifiedoftheerror.However,

ifyoumistypethenameofapropertyyouwanttoprojectorqueryfor,youwill

not get an error but will get an empty result instead. For example, try the

projectioncriteriapropertynameas{"blah":1}.Ifyoudothis,youwillnotget

anerrorbutwillgetanemptyresultset.

KeepinmindthatMongoDBisaschema-lessdatabaseanditassumesthat

the“blah”propertycouldbethereinthefuture,butitjustisnotthererightnow.

Propertiescancomeandgoinaschema-lessdocument-baseddatabase.

6.4IndexingPolicy

Youcould writecodethat usestheMongoDBAPIthat runsto createyour

neededindexes.Myapproachistonotputthingsinthecodethatareone-time

configurations. I will instead prefer to use the Compass application to create

indexes. You could also use the mongo shell to run a command to create the

index.

For the NewsWatcher application, you can imagine that you would have a

queryintheservicelayerthatwilllookupauserbytheiremail.Youwouldwant

toaddaspecificindexforthatbydoingthefollowing:

1. OpentheCompassapplication,clickthenewswatchedbdatabase.

2. Clickthenewswatchercollection.

3. ClicktheINDEXEStabthenclickCREATEINDEX.

4. Givetheindexaname.

5. Select“email”asthepropertyand“1(asc)”.

6. SelectthecheckboxforCreateuniqueindex.

7. Select the check box for Partial Filter Expression and enter

“{email:{$exists:true}}”

8. ClickCreate.

You check Create unique index as you don’t want email addresses

duplicated across users. This is a way to uniquely identify an account for a

person. You have also set up what can be called a “sparse index” using the

PartialFilterExpression.Thismeansthatdocumentsthatdon’thavethe“email”

property will not be used in the index. This will give you lower storage

requirementsandofferbetterperformancewiththeindexmaintenance.

Figure27-Creatinganewindex

6.5MovingOn

Thiscompletestheworktogetthedatalayerupandrunning.Youcansee

thatitwasallaboutsettingupyourconfigurationthroughtheAtlasmanagement

portalandtheCompassapp.Youdidnotneedtowriteanycodeyet.Thismeans

that you are postponing the writing of any Node.js JavaScript server-side

functionalityuntilyouworkontheservicelayer.

Testing

For the NewsWatcher application, the service layer is actually the proving

ground for the data layer. The service layer will connect directly to the

MongoDB collection and perform CRUD operations. There will be functional

testsputin placeto provethat thedata modelworks. Youwillalsobe ableto

takecareofthenuancesthatgoalongwiththedatalayer,suchasperformance

tuningandconcurrencyissues.

In reality, the best way to develop software is to work on it in terms of

verticalslicesoffunctionality.Thismeansthatforanyfeaturesyouhavethought

up,youwouldimplementitinallthreearchitecturallayersatonce.

Chapter7:DevOpsforMongoDB

In this chapter, I will go over some of the operational responsibilities for

managing a MongoDB database. For example, with NewsWatcher, you know

that the data layer stores user accounts and news stories. You can think about

whatconcernsyouwouldhavewiththat.DailyDevOpsworkwill involvethe

monitoringofthedatabase.YoucantakealookatwhattheAtlasmanagement

portalwillletyoumonitor.

Youwillwanttomakesurethatdataaccessissecureandperformant.You

can set up replication, sharding, and indexes. Once those are set, you should

leavethemaloneuntilsomechangecomesalongthatcausesyoutotweakthem

foraspecificreason.

7.1 Monitoring through the Atlas

ManagementPortal

Oneviewyoucanlookatistheviewofmachinesinyourdatabasereplica

set.Hereisascreenshotthatshowsallthemachinesinareplicaset(primaryand

twosecondaries):

Figure28-Atlasmanagementportalserverview

Note:ThefreetierofferingislimitedinwhatitoffersintheAtlasportal.For

example,thereisaDataExplorerwiththepaidtier.TheMetricspageforapaid

tierhasalotmoreinformationavailablelike-ShardedClusterMetrics,Replica

SetMetrics(moremetrics),Real-TimeTab,StatusTab,HardwareTab,DBStats

TabandChartControls.

Telemetrycharts

You can drill down further into the performance metrics of each of these

machines by clicking on them. There are charts provided in the Atlas

management portal to show you the server utilization numbers. The real-time

telemetryvalueswillletyouknowthingslikehowmuchstorageyouhaveused

up.

Here is an image showing the Atlas management portal monitoring page

withthetelemetrythatisshownbydefault.Inareplicaset,youwillhavemore

than one machine, so you have to pick the primary or one of the secondary

machinesifyouwanttoseetheirtelemetryseparately.

Figure29-Atlasmanagementportalmetricspage

Telemetryalerting

You probably want to be aware of how much storage is left for your

database. You could set up an alert to notify you when you are reaching this

limit.Youmightalsobeconcernedaboutmachineperformanceandsetupsome

alerts around specific performance measurements. If you see your machine

performancedegrading,thenyoucanshifttoamorecapableconfigurationwith

thePaaSofferingsofMongoDBInc.

IfyougointoyourAtlasportal,youcanselectAlertsfromthemenuonthe

left.Thenyoucanselecttheappropriatetabfromthere.Forexample,thereisa

tabtoviewandacknowledgealertsthathavebeentriggered.

ClickontheAlertSettings tabto seewhat alertsyou haveby default and

add any you need. In the figure below, you can see that there is an alert that

triggerswhenyouhavereached90%ofyourstoragecapacity.

Figure30-AtlasAlertConfigurationpage

Youcanselectfromawidechoiceofpossibilitiesforhowyougetnotifiedof

an alert being triggered. Click on the Add button to see these. The selections

include – Atlas User, Email, SMS, HipChat, Slack, Flowdock, PagerDuty, and

Datadog.

7.2TheBlameGame

Onceyourdataissecureandhasbeentunedforthebestperformance,you

really don’t need much in the way of day-to-day care anymore. Believe me

though when I say this – your potential troubles are not over by any means.

Frommyexperience,youwillbespendingyourtimecaringfortheintegrityof

the data as much as anything else. This is especially true if there are a lot of

othersystemsintegratingwithyoursthattouchthedataatsomepoint.

Unfortunately, every time someone sees a data corruption problem, they

come to blame whoever is in charge of the DBMS. You will hopefully have

confidencethatmostofthetimetheaccusationsareunwarranted,andyouwill

be able to track the problem down to some supporting system. For example,

some external system that is feeding you data may suddenly have missing,

intentionally altered, or corrupt data. It is a good idea to put data validation

measuresintoplaceatallthepointsofintegration.

Youwouldalsobewisetoputsomehandyscriptsinplacetoallowyouto

diagnoseissuesandfixthem.Forexample,youmightneedtorecoverdatafrom

abackupsnapshotorreimportdatainbulkfromadependentsystem.

7.3BackupandRecovery

Thegoodnewsisthatyoudon’thavetoworryaboutdisasterrecovery.The

bad news is that you have to worry about disaster recovery. It all depends on

whatyourdefinitionisofa“disaster”.

Withareplicasetinplace,MongoDBstoresmultiplecopiesofyourdatathat

arealwaysinsyncwiththeprimaryserver.Thismeansthat,withinaregion,you

haveredundancyincaseofnetworkordrivefailures.Thisisthecasewiththe

differentAWSavailabilityzonesthateachEC2serverisinforyourreplicaset.

You have this set up for you through the PaaS plan you select and don’t

necessarilyhavetodealwithitdirectlyyourself.

This replication means that your data is safe from drive failures, machine

reboots,poweroutages,networkoutages,andsuch.Ifthedrivethattheprimary

copy of your MongoDB database is on goes bad, you are covered. MongoDB

and AWS will take care of rotating this drive out and moving you to a new

primarydriveandaddinganewreplacementbackup.

Youstillneedbackups

Datareplicationis notthesame asperforming adata backup.Just because

youhavereplicationdoesnotmeanyouareprotectedfromsomehowlosingor

mangling your own data by mistake. It is a good idea to institute a backup

processtoperiodicallystoreasnapshotofallyourdata.Thatwayyouareableto

recoverfrominadvertentcorruptionorlossofyourdata.

Backupsare useful in many scenarios. For example, you might have some

bugthatwasintroducedinyourcodethatcausesallyourdatatogetcorrupted.

You then need to roll back to the database copy you had before the data was

corrupted.

Youcould,forexample,writesomecodetocopydatafromoneMongoDB

regionintoanotherregionandkeepthatasabackup.

You could also save a collection as a file for safe-keeping on some local

machineyouhave,orplaceitinEBSorS3storageinacompressedform.Then

youcandoarestoreprogrammaticallyoruseatooltoimporteverythingfrom

yoursnapshot.YoucanexportaJSONfileandstoreityourselfifyoudon’twant

tospend the moneyon database collectionsor otheronline storage beingused

forbackups.

TheAtlasmanagementportalhasaBackupselectionontheleft-handside

thatletsyoucreateanimmediatebackup,ortoscheduleatimeeachdayforone

to automatically happen. There is a pre-determined retention policy for each

specifictime-relatedsnapshot. The coreMongoDB projecthas backup utilities

youcouldalsousetoperformdatabackups.

PARTII:TheServiceLayer(Node.js)

Parttwoofthisbookwillteachyouwhataservicelayeris.Youwillcreate

anHTTP/RestAPIthatinterfacestothedatalayer.Thiswillsetthingsuptobe

preparedforthedevelopmentof thepresentationlayerapplicationthattalksto

the service layer. Node.js/Express.js and JavaScript are the technologies of

choiceforthisservicelayer.

Therearemanydecisionsthatgointocreatingaservicelayer.Thefirstthing

to design, is what type of interface is needed over the data. This involves

separatingoutthedifferenttypesofdatathatyourRESTinterfacewillexpose.

ThisrequiresyoutothinkabouttheJSONpayloadsthatgettransferredbackand

forthforeachrequest.

You will learn how to use a Node module to call into the MongoDB data

layer developed in the first part of this book. At the end of this part, the

applicationwillbefullyfunctionalandreadytointegratewiththepresentation

layer.

Thisbook will be using Amazon Web Services (AWS) to host the Node.js

application.

Theextremelyimportanttopicoftestingwillbecovered,andyouwilllearn

how to use the Mocha test framework to run your tests. You will learn about

functionalaswellasperformanceloadtesting.

Inordertogivefullcoveragetothetopic,Iwillalsodiscusswhatitmeansto

setupallaspectsoftheday-to-dayoperationsofNode.jsforactualdatacenter

operationsmanagement.YouwilllearnhowtomanageaPaaSenvironmentand

howtododebuggingofissues.Securitywillbeanimportanttopicthatisalso

covered.

Note:ManypeoplerefertoNode.jssimplyasNode,andIoftendothesame.

Chapter8:Fundamentals

This chapter presents the fundamental concepts of the middle-tier of the

three-tierapplicationarchitecturethatisbeingoutlinedforyouinthisbook.You

willlearnwhatthemiddle-tieristypicallycomposedof.Icanthengetintothe

specificsandshowyouhowNode.jscanserveasamiddle-tierservicelayer.

8.1DefinitionoftheServiceLayer

The service layer provides the core capabilities of a three-tier architecture.

Thewholeideaofaservicelayeristobuildanabstractionlayeroverbusiness

logicanddataaccess.

Ifyoureallysimplifydowntheconceptsofathree-tierarchitecture,youcan

saythisaboutthelowerandupperlayers-thelowerdatalayerjuststoresdata

andtheupperpresentationlayerjustdisplaystheUserInterface.Thatleavesthe

middle-tierservicelayertodoalltherestofthework.Inmostapplications,you

willcertainly find morecode in the servicelayer than in theother layers. The

followingdiagramshowsthissimpleview:

Figure31-Simplifiedthree-tierarchitecturediagram

Itwouldnotbereasonabletohavethepresentationlayerhandlethebusiness

workflowlogic.Youalsodonotwanttoexposebusinessworkflowsinthedata

layer.Thedatalayershouldbekeptassimpleaspossibleandshouldonlyhandle

theCRUDoperationsandperhapsmoredifficultdatatransactionlogic.Youwill

normallyfindthemorecomplexbusinessservicescodeintheserviceslayer.

Note: Some architectures split the middle-tier out into a services and a

businesslayer.Thesetwoconceptsarecombinedinthisbook.Youwillfindthat

alltheneededfunctionalityofthislayercanbeaccomplishedwithinthesingle

technologyframeworkofNode.js,alongwiththeuseofnumerousnpmpackages.

Acontractofinteraction

A service layer can actually be created with no particular UI in mind. For

example,therearemajorcompaniesthatexpose theirAPIssothatanyone can

interface with their backend and write their own UI. Companies like eBay,

Twitter, and Facebook have been successful at this. For example, you can

interactwiththeeBayAPItobidonitems.

InthecaseoftheNewsWatchersampleapplication,thereisjustonesingle

UIthatIwrote,butanyoneelsecouldwriteadifferentUIontopofmyexposed

RestAPI.

Regardless of whether you are tied to one single front-end UI, or if your

servicelayerisopentoallowmanyapplicationstoconnect,youneedtothinkin

terms of a strict contract of interaction. This means you must define the

connectionroutesupfrontandtheJSONmessagesthatarerequired.

I will not be using any specific connection standard like you see with

standardssuch as SOAP.You canexplore things like Swagger on yourown if

youareinterestedinmakingyourAPIgenerallyavailabletopeopleinaneasyto

consumeandformalway.YoucanalsoexploreanAWSofferingknownasthe

APIGateway.ThiscanbeusedtosurfaceaformalAPIcontractthatsitsinfront

ofyourNode.jsservicelayer.

Anabstractionlayer

Aservicelayerisbuiltinawaythatabstractsawaythecomplexitiesthatgo

oninthebackend.Thiswillshieldthepresentationlayerclient-sidecodefrom

any tight coupling. The client is also protected from any backend rework. In

many cases, the client doesn’t even need to make any changes when backend

codeisrewritten.

Itmaybe,thatasinglecalltotheservicelayerresultsinaseriesofbackend

callsthatareeachprocessedinwhatcanbecalledaworkflow.Thiscoordination

fallssquarelyintheservicelayerinordertohidethecomplexity.Themultiple

backendservicescompriseyouroverallarchitectureandcanbeunifiedthrougha

singleNode.jsentrypoint.

For example, you might have one service that does all of the storage and

retrievalofall useraccount information.Another servicemight containbilling

account information, and yet another might deal with order information. You

shouldneverbuildonesinglemonolithicservicethatdoeseverything.Takethe

timetosplitupyourplatformsintodiscreetservices.Eachofthesewillservea

role, be self-contained, and operate independently. Research what is called a

microservicesarchitecture.

Thisdoesnotmeanthatexternalclientsneedtoauthenticateandconnectto

each of the microservices individually. In other words, you might want your

presentation layer to only have one single service layer entry point that

coordinates calls to other independent microservices that each exists as

autonomousservices. Youwillactually be gratefulyou have donethis, as you

canmakerathersignificantchangesinthelowerlayersandminimizethecode

thathastochangeintheclient.

Your backend might already consist of “legacy” systems that were not

written with Node. These backend systems might be written in different

languagesandberunningonyourownproprietary,on-premiseplatforms.Inthis

case,youcandecidetowriteagatewayservicelayerwithNodeandhavethat

code interface to your backend systems. Node is great for routing and

orchestration.Itcanworktoprocessrequestsasynchronouslywithahighdegree

ofconcurrency.AWSoffersanAPIGatewaythatyoucanalsoinvestigate.

AllorpartofyoursystemscanhavetheirownNode.jsinterfaces.Itisupto

you to decide what is worth your time and investment to do. The following

diagramillustratesthegatewaylayeringthatcouldexistinaserviceslayer:

Figure32-Servicelayergatewayconcept

If you decide you need to replace the billing system, the gateway service

providestheabstractionandprotectstheclientfromanychanges.

Servicelayerplanning

Knowing what operations and workflows are needed is the first step in

creatingaservicelayer.Usethefollowingquestionstohelpyoudeterminethe

designofyourservicelayer:

Whatoperationsandworkflowsareneeded?

Whatareyoursecurityandprivacyrequirements?

Howarepeopleauthenticatedandauthorized?

Is there a need for a pub/sub notification system to deliver push

notifications?

Isthereanytypeofdomain-specificconfigurationrequired?

Is programmatic resource management required? Scripting or

templates?

Arealldatainteractionsencrypted?

Aremultiplesystemsgoingtocallthis?Wouldamessagecontract

schemabeappropriate?

What validation can be made at the interface to not let anything

invalidin?

Whatmeaningfulerrorsneedtobereturned?

Doyouneeduserrolesandaccesscontrol?

Doyouneedcachingontopofyourdatabaselayer?

Is there any periodic processing of the data? Is it periodic in the

backgroundorreal-time?Inbatches?

Willyou need to queue up workand have workers process work

asynchronously?

WhataretheSLArequirementsforalloperations?

Whataretheaccessvolumeandtheratespertimeperiod?

Willtherebeburstsofactivityorisaccessevenlydistributed?

The answers to these questions should be carefully considered. Consult

experts along the way before you roll anything out into a full production

environment.

8.2IntroducingNode.js

The simplest way to describe Node.js, is to state that it is a runtime that

executesJavaScriptcodethatyouprovideit.YoumightthinkIjustdescribeda

browserenvironmentforyou.Afterall,thisiswhattheChromebrowsercando.

The Chrome browser has a JavaScript engine called V8 that is used for

executingclient-sideJavaScriptcode.

Node.js, however, is more typically running on the server-side. To

accomplish this, someone (Ryan Dahl) actually took the same V8 engine

mentioned,andmadeuseofitinaserver-sideprocessandthenaddedadditional

functionalitythatwouldmakesensetohaverunninginservercode.

You can think of Node as an abstraction layer on top of your operating

systemsoastomakeyourcoderunasplatformindependentcodeandgetmany

of the capabilities that an operating system provides (file system, processes,

networketc.).Node.jsprovidestheoverallruntimeexecutionenvironment.

Platformindependence

Letme putit this way- you could goand write anapplication inC++ for

Windowsthatimplementsawebservicethatlistenstoandrespondstorequests

andinteractswiththefilesystem.Butwhatwould youhavetodototake that

application and make it run on Linux? You would have to port it of course,

which requires a re-write of that C++ code to use operating system libraries

foundonLinuxmachines!

To get that to run on a Linux OS, you would port your code to use those

systemcalls that are availableon Linux. This image shows thatyou would be

writingyourcodeoverandoverforeachplatform.

Figure33-CodeportingacrossfromOStoOS

Nodeletsyouwriteyourapplicationcodeonce,andthenNodehandlesthe

lower level porting of Operating System level calls for you. Node.js acts to

abstractawaytheplatformOScapabilities.Notonlythat,butNodeallowsyou

towriteallyourcodeinJavaScript!

Figure34-Writingcodeonce

Node.js is an open-source project and has been ported to run on many

different operating systems. Much of the core code of Node.js is written in

C/C++ to enable native integration with underlying operating systems and

achievethefastestpossibleperformance.ItalsoutilizestheGoogleV8engineto

execute JavaScript. V8 actually compiles client JavaScript to native machine

code,suchasforx86machinearchitectures,forfasterexecution.

Node.jsiswell-suitedfornetwork-basedI/Oapplications.UsingNode,itis

extremelysimpletopiecetogetherawebserversimilartoIISorApache.You

can easily set up a web service to expose an HTTP/Rest API that works with

JSONpayloads.Nodereallyfulfillsalotofpurposesthatallowittosatisfyall

therequirementsofamiddle-tierservicelayer.

ExtensibilityofNode

TherealpowerofNodecomesthroughitsextensibility.Nodewaswrittento

providethecoreruntimeof execution,schedulingandnotificationcapabilities.

Itsfunctionality is then greatlyincreased through the manyextension modules

written for it. For example, there are modules for functionality such as

WebSockets, data caching, database accessing, asynchronous processing,

authentication,andmanyothers.

Nodeiswidelyadopted,andpeopleareconstantlyimprovingitandcreating

newmodulesforit.Thereisalargecommunityofdevelopersthathasgenerated

many extremely useful modules to give rich functionality to your application.

Since the patterns of using these modules are all very similar, it is extremely

easytoconsumethemwithoutanylearningcurve.

JavaScriptbliss

Since Node.js executes JavaScript, there is a consistency throughout the

applicationstackthatyouarebuilding.JavaScriptdesignpatternsareutilizedin

theconstructionofmodulesthatbecometheself-containedcomponentsthatare

availabletoconsume.Youcandownloadandintegratethoseofferedbyothers,

aswellascreateyourown.

There are also unit-testing frameworks that work with the JavaScript

languagethatyouwillusetotestyourNode.jsservicelayer.

JavaScriptisnowutilizedasmorethanjustaclient-sidescriptinglanguage.

Itisnowaformidableserver-sidelanguage,asimplementedwithNode.js.

Note:LestIgetaslewofemailsaccusingmeoflivinginafairytale,Iwill

make a brief comment on the sensibility of using JavaScript in enterprise

applications. It is true that JavaScript can be a challenge with respect to

deliveringonquality.However,withproperdesignpatternsandtesting,youcan

today create large enterprises services of superb quality with Node.js and

JavaScript. Many large companies have already done so. Perhaps as the

JavaScript language evolves more features will convince the skeptics that it is

heretostay.

8.3BasicConceptsofProgrammingNode

AsimpleNodeprogramthatyoucanwritewouldbeaprogramthatoutputs

texttotheterminalprocesswindow.Thefollowing isanexampleofwhatthis

wouldlooklike:

console.log("HelloWorld");

Youcantypethisintoafile,saveittodisk,andthenhavetheNodeprocess

execute it. If you had Node installed, you could open a command prompt and

typethefollowing,substitutingthenameofyourfilefor<filename>.

node<filename>.js

If you haven’t already done so, you should go ahead and install Node on

your machine at this time. Go to https://nodejs.org/. You will see a download

link labeled “LTS” and another labeled “Current”. You want the LTS version

(LongTermSupport),asthatisthestableone.Theotheroneisfromthelatest

codeunderdevelopmentandhasnotbeensufficientlyproven.

Go ahead and create a file named server.js, place the console.log("Hello

World");lineinitandrunnodeasshownwiththisfileasanargument.Youneed

tobeinthesamedirectoryasyourfile.

Youwillalsonoticethattheprocessdoesnotstayrunning.Inthiscase,once

yourcodeinyourfilehasbeenexecuted,theNodeprocessexits.Thisdoesnot

havetobethecase.IwillsoonexplainwhatwouldcauseaNodeapplicationto

keep running so it can continually perform server-side processing such as the

processingofwebrequests.

TheREPL

Node has several different options to control how it runs. If you were to

leaveofftheJavaScriptfile,NodewoulddefaulttowhatiscalledREPLmode.

REPL is the mode where you get a prompt and can enter JavaScript to be

executedasyoutypeitin.REPLstandsforReadEvaluatePrintLoopandisa

common thing for execution frameworks to provide. For example, MongoDB

providessomethingsimilar,calledtheMongoShell.

Inthisbook,youwillnotbeusingtheREPLandonlyneedtobeconcerned

with the main means of invoking Node as shown already, passing your

JavaScriptfileasanargument.

NodeexecutesJavaScript

Hereisanotherfiletotry.ThisoneillustratesabitmorecodethatNodecan

execute.Runningthiswillresultin“HITHERE343”beingdisplayed:

varx=7;

vars="Hithere";

functionblah(num,str){

if(num==0){

return"Can'tdothat";

}

returnstr.toUpperCase()+""+Math.pow(num,3);

}

varresult=blah(x,s);

console.log(result);

This illustrates some basic JavaScript language capabilities. If you look at

theJavaScriptspecification,youcouldseemoreofwhatispossible.Youhave

datatypes,operators,structuredprogramming,logiccontrol,built-inobjectsand

muchmore.

Ifyouhaveprogrammedbrowserscriptsbefore,becarefulaboutwhatyou

assume is available in JavaScript. For example, there is a function named

setTimeout()thatyoumightassumeisapartofthecorelanguageofJavaScript.

This is not the case. This is where browser implementations have added

functionalitytoJavaScript.

ThesetTimeout()functionisindeedprovided,butitisimplementedthrough

theNode.jslayersandnottheJavaScriptlanguageitself.Thisisalsotrueforthe

console.log()function.

As was mentioned, the Node process will run and exit upon execution of

yourlinesofcode.ThisisbecauseNode.jswillonlyrunwhileitknowsithas

codetoexecute.Tryrunningthefollowingcode:

while(true){

console.log("HelloWorld");

}

To stop the Node.js process, you will have to press <Ctrl> C on your

keyboard, or close the window. Later, you will see some code that requires

Node.jstorunforeverbecauseithasbeensetuptorespondtoeventsthatcould

perpetuallyhappen.

Usingbuilt-inmodules

As was explained, Node extends the basic capabilities of JavaScript by

providingasetofbuilt-inmodules.Therearequiteafewofthem,andyoucan

reviewthemifyougotohttps://nodejs.org/en/docs/.ClickonanAPIversionon

thelefttoseethelistofmodules.YouwillseethingsonthelistsuchasHTTP,

Net,OS,Crypto,FileSystem,Console,andTimers.

You can now see how to use the Node.js provided functions such as

console.log()andsetTimeout().Hereissomesamplecodethattakesadvantage

oftheseaddedcapabilities:

setTimeout(function(){

console.log('World!');

},1000)

console.log('Hello');

IfyouarenotfamiliarwithhowsetTimeout()works,youhavetobeaware

thatthisschedulesacallbackfunctiontorunsomenumberofmillisecondsinthe

future. Thus, the string Hello prints first and then one second later you see

World!.

Let’snowlookathoweasyitistocreateanHTTPWebserverwithcodethat

Nodewouldexecute. Thefollowingexample isallthe codeneeded ifyouuse

thebuilt-inHTTPmodule:

varhttp=require('http');

varserver=http.createServer(function(request,response){

response.writeHead(200,{"Content-Type":"text/plain"});

response.end("HelloWorld\n");

});

server.listen(3000);

IfyouweretoexecutethiswithNodeonyourlocalmachine,youcouldthen

openabrowserandnavigatetohttp://localhost:3000/andseeyourHelloWorld

messageappear.

The only non-obvious line of code is the very first one. This is a function

definedinNode.jsthatyoucalltoloadtheHTTPmodulethatprovidestheweb

servercapability.ThosewhowritebrowserscriptsinJavaScriptusean‘import’

statementinstead(seeNode10docsforusage).

Node uses the concept of modules as its extensibility mechanism. The

require()functionsimplyreturnsanobjectwithfunctionsandpropertiesplaced

on it that are specific to that module for you to use. In this case, you call

require('http')and get an object, and then use the createServer() function from

that object. Some things like the setTimeout() function are made available

globallywithoutyouneedingarequire()statement.

Usingexternalmodules

In some programming languages and runtimes, you mostly rely on what

comes built in. For example, this is the case with the combination of C# and

.Net.InNode,however,youwillconstantlybelookingforexternalmodulesto

add to your application to give you many of your capabilities. As a matter of

fact,sincetherearesomanyoftheseexternalmodulesprovidedonline,youwill

need to become good at searching for something and then determine what the

bestoptionis.

There is a package manager site you can go to for searching and

downloadingmodules.Gotohttps://www.npmjs.com/totakealook.Beaware

thatsomeofwhatisfoundinNPMarecodemodulesyoucanuse,whileother

downloadsareactuallytoolsyoucanrun,suchasthetestingtoolcalledmocha.

WhentheNPMmodulesareutilized,thelayersofcodenowlookasfollows:

Figure35-AddingNPMmodulestothearchitecture

Thisgivesyouthehighest-levelviewofthedifferentblocksofcode.

Note: Regular Node modules are written in JavaScript and distributed for

otherstouse.Thereis,however,awaytowriteamoduleinC++thatisreferred

to as an add-on. If you do that, you could get better performance (more

sustainedcomputeintensivecode)andalsoaccesstooperatingsystemAPIsthat

arenativetoamachinethatmightonlybeavailableinC++libraries.N-APIis

anewer capability providedby Node for buildingnative Add-ons. Learning to

create Addons is not necessary for most projects and the topic will not be

coveredinthisbook.

To use external modules from NPM, you need to have them installed

alongsideyourownJavaScriptfile.TherearetwostepsneededtouseanNPM

codemodule:

1. Createafilenamedpackage.json.

2. Runthecommand“npminstall<module>--save”foreachmodule

youwanttouse.

Running the npm install command will actually add a line in your

package.jsonfile. The veryfirst time you run itin a directory,a folder named

node_modules is created. If you look in that folder, you will see all of the

modulecode.WhenyourunNodewithcodethatrequiresoneofthesemodules,

Nodewillbeabletofindthem.

You can try the steps by creating the package.json file with the following

linesinit:

{

"name":"test",

"dependencies":{

}

Nowrunthefollowingcommand:

npminstallasync--save

Afterrunningthis,anode_modulesfolderiscreated,withtheasyncmodule

filesinstalled.Yourpackage.jsonfilewilllookasfollows:

{

"name":"test",

"dependencies":{

"async":"^1.5.2"

}

Thereisaconventioninthepackage.jsonfileforlistingmodules.Thename

islistedfollowedbytheversionyoudesire.Youcanspecifyanexactversion,or

you can specify just the major number and have NPM get the latest minor

version.The‘^’characterforourusageoftheasyncmodulemeansthatifany

timeyourefreshyourusagewithan“npminstall”commandyouwouldbringin

anyminoror patchupdates.For ourexampleabove, thatwouldnot bringin a

2.x.xversion,asthatwouldbeamajorversionupdate.

If you add dependency modules in your package.json file by hand and

specified the exact version, you could then run the “npm install” command.

NPMwillinstallthelatestversionforwhatversionnumberingyouspecify.The

followingcommandiswhatyourunifyoueditpackage.jsonfirstandthenwant

toinstallthemodulesyouspecified:

npminstall

Itistypicalfordeveloperstostickwithaknowngoodworkingversionfor

eachoftheirmodulesandnotupdatetoanynewmajorversionsevenwhenthey

become available. When you feel you need some new capability or security

patch of a newer version, then do an update and do extensive testing of

everythingagain.Majorversionshavenewfunctionality.

Hereisanothercodesampleyoucanrun.Itmakesuseoftheasyncmodule

thatwasinstalled.Putthiscodeinyourserver.jsfile:

varasync=require('async');

varfs=require('fs');

async.eachSeries(['package.json','server.js'],function(file,callback){

console.log('Readingfile'+file);

fs.readFile(file,'utf8',functionread(err,data){

console.log(data);

callback();

});

},function(err){

if(err){

console.log('Afilefailedtoload');

}else{

console.log('Allfileshavebeensuccessfullyread');

}

});

Youwilllearnmoreabouttheasyncmodulelater.Thisisbasicallyusingthe

async module capability to sequence through an array of values and do what

processingyouwantoneachentrysequentially.

Notethatyouareusingthefs module.Youdon’tneedto runaninstall,or

evenlistthefsmoduleinthe package.jsonfile.Thisisbecausethismodule is

partofNode.js,butyoustillneedtherequirestatement.Ifyounowrun"node

server.js",youwillseethecontentsofyourfilesprintedout.

Note: Deployment of your Node.js application is easy. You can just copy

everything,includingthenode_modulesdirectory,toamachine.Whenyouusea

PaaSenvironmentinstall,suchaswithAWSElasticBeanstalk,youdon’tneedto

copythenode_modulesdirectory.AWSwillrunthenpminstallfor you.Ifyou

want version numbers locked, set the specific versions or use what is called a

shrinkwrapfileoruseafeatureofNPM5forthat.

Callbacksandconcurrentprocessing

To start with, you need to understand that there is just one main thread of

executioninyourprogram.Thisisthethreadthatstartsupyourapplicationand

begins execution of your JavaScript code. From there, Node.js sends all your

codetotheV8JavaScriptVMengine,andOSportedcallstobeginitsexecution.

EverythingatthehighestlevelofyourJavaScriptplacesprocessingtimeon

asinglethread.Onlyonethingcanhappenthereatatime,sowritecodethatis

notcomputeintensive.

Note:VMstandsforVirtualMachineandisaconceptthatV8usestoisolate

JavaScriptexecution.Don’tconfusethisdefinitionofaVMwithaVMthatyou

findhostedinAWSorMicrosoftAzure.

YouhavealreadyseenNodecodethatusesthecallbackstyleofcoding.This

styleisprevalentineverythingyoudoinNode.TheNode.jslibraryprovidesfor

thesenon-blockingasynchronouscallbacks.Yourcodeneverblocks,butreturns

immediatelyandthenatsomelatertime,thecallbackfunctionisexecuted.This

givesyoutheconcurrentexecutioncapabilitythatNodeisfamousfor.

Note: There are other mechanisms for doing asynchronous code such as

usingPromisesandAsync/Await.Thesearealsoverypopularandarebasically

asyntaxpreference.Async/Awaitwillnotbecoveredinthisbook.

Codeexecutionflow

Thecodeexecutionpathisinterestingtotracethrough.Iwillnowexplaina

bitofhowthisallworks.Lookatthefollowingcodethatwillbeusedtohelp

understandtheexecutionflowforyourJavaScriptcode:

varx=7;

vars="Hithere";

varfs=require('fs');

functionblah(n,s){

if(n==0){return"No";}

returns.toUpperCase()+""+Math.pow(n,3);

}

console.log(blah(x,s));

fs.readFile('package.json','utf8',function(err,data){

console.log(data);

});

Ifthiscodeisinyourserver.jsfileandyouexecuteitonthecommandlineas

nodeserver.jsyourexecutionlooksasshowninfigure36.Youcanseethecode

boundarycrossings.Youcanalsoseethatalmosteverythingisnon-blocking.At

alowerlevel,thereisathreadinNodethatdoesendupbeingblocked,butthis

thread does not affect you at all. Your code still has the callback that is

asynchronouslycalled,soyouarenotblockedbyit.

Inthe illustration, the execution time moves leftto right. I have illustrated

theboundarybetweenyourmainJavaScriptthreadandtheNode.jsframework

with Libuv (see a later chapter for more information). The upper part is your

codebeingexecutedintheV8VM.

Figure36-Codeexecutionflow

Followlefttorightinthefigureaboveandyoucanseehoweachbitofcode

is run. To start with, the line of code that does the require() will block code

execution until it completes. The console.log() call does not block. The lower

layerwillasynchronouslyprintoutthevalue.Thefunctionblah()iscalledandin

that, the JavaScript Math.pow() object function executes synchronously in V8

andnotintheLibuvlayer.

Thefs.readFile()callusesanasynchronouscallbacktokeepyourupperlayer

codenon-blocking.YoucanseethethirdparametertothereadFile()functionisa

functioncallback.TheNode.jsframeworkstartsexecutingreadFile()forthefirst

bitofcode,buttheNode.jscodeisjustcallingintoLibuvtohandofftherequest

to be executed on its thread pool. This then immediately returns, and your

executioncontinuesinthecode.

TheLibuvexecution threadfor thefileI/O eventuallyreturnsand thenthe

callbackfunctiongetscalledandrunsonthemainthread.Youcannotgetaccess

tothethreadpoolprocessingdirectlyfromyourJavaScriptcode.

FilesystemcallsgotothethreadpoolofLibuv.Networkcallsareprocessed

differentlythanfilesystemcallsaswillbeexplainedlater.Ineithercase,Libuv

doesall the workfor you to makethings work acrossplatforms and in anon-

blockingway.

ContinuousprocessingwithNode

AnotherconceptthatyouneedtobeintroducedtoisthatofhowNodecan

berunninginacontinuousprocessingloop.Asshown,thepreviouscodesample

runstocompletionandthentheNodeprocessexits.ThisisbecauseNodeknows

ifthereisanymoreworktoexecuteandifthereisn’tany,itjustexits.

You can understand that there are certain modules you can use that will

basicallykeeptheNodeprocessrunningforever.Ifthisisthecase,youcanstop

theNodeprocessasyouwouldnormallydoonyouroperatingsystem.

Yousawcodeearlierthathadawhileloopthatneverhadanywaytoexit.

Thatexamplewouldbealittleodd,sinceitrunsallofthetime,andcompletely

blocksthesingleprocessingthread.Amorereasonablepieceofcodewouldbe

somethingthatusesanintervaltimertodosomeperiodiccomputation.Hereis

somesimplecodethatkeepsNoderunning:

setInterval(function(){

console.log('Helloagain!');

},5000)

console.log('HelloWorld!');

This is actually something similar to what the NewsWatcher code does to

periodically look for news stories. NewsWatcher will need to run some

processingonaperiodicbasis.

Anotherthingthatwillkeepyourprocessrunningforeverwouldbetheuse

ofmodulessuchasHTTP,NetorExpress.Whenyousetupthecodetolistenfor

TCP connections and listen on a socket, you set up code that will run in the

lowerlevelofLibuv.Takethefollowingexamplethatwasusedbefore:

varhttp=require('http');

varserver=http.createServer(function(request,response){

response.writeHead(200,{"Content-Type":"text/plain"});

response.end("HelloWorld\n");

});

server.listen(3000);

What happens here, is that Libuv is set up to use the low-level OS socket

capabilitiesto listenand respond to incoming connections and requests. Libuv

then has a loop to respond to any of these events. When they happen, your

callback code can run, such as the one above. The main Node.js process loop

actuallycheckswithLibuvtoseeifitneedstoberunningbecauseofworkithas

initsqueue,orislisteningfor.Ifso,thentheprocessiskeptalive.

Note: the Libuv thread pool is not involved in socket listening. This is

becausethelow-levelOScapabilitieshandletheasyncnon-blockingprocessing

andgeneratesthenotificationeventsuponcompletion.

8.4Node.jsModuleDesign

Node itself is composed of various modules that run as part of the core

service.ModulesarewhatenableallthefunctionalityinNode,besideswhatis

provided with the JavaScript language. The simple “Hello World” example

demonstrated this. That example made use of the console module. Other

modules, such as the express module, are third-party modules you install, to

bring in additional functionality. As mentioned, you use the Node Package

Manager(NPM)togetallyourexternalmodulesinstalledonyourmachineand

thenusearequire()statementtousethemincode.

You will write many of your own project code as modules for your own

consumption.Ifyouarereallyambitious,youmightevenwanttowriteamodule

and make it available as a download from the NPM repository for others to

benefitfrom.

I will now show you how a module is constructed, and you will see how

Nodeexposesthefunctionalityofamodule.Youdon’tnecessarilyneedtoknow

howthisworksinternaltoNode.js,butIincludethisinformationforthosethat

arecurious.

AmodulecanreturnaJavaScriptobject

Modules contain JavaScript code that is typically set up as an object. The

object is then exposed in a special way, so that a client can make use of it

throughtherequire()functionasshowninapreviousexample.Node.jsdoesthe

worktotakeyourmodulecodeandsurfaceitthroughitsinternalexposurefor

other code to call. Node keeps track of all the loaded modules and manages

loading, configuring, running, and caching of the modules. All require

statementsthroughoutyourcodewillreturnthesameobject(i.e.HTTP)asitis

cachedonceandusedforeveryrequire(singletonpattern).

Allyoureallyneedtoknowisthatmodulecodeisexposedthroughaspecial

Nodeobject named exports. The exports object is created for you by Node in

eachandeverymodulefile.Then,whencodecallstherequire()function,Node

returnstheexportsobjectwithwhateverfunctionalitywasplacedinit,suchasa

function. The following example shows a simple module you could write that

exposesasinglefunction:

//mymodule.jsthatholdsyourmodulecode

module.exports.welcome=function(name){

console.log(“Hi”+name);

}

AsImentioned,youprovidefunctionsandpropertiesinamodulefile.These

arethenexposedoutsideofthatfile.Youcanaddpropertiestotheexportsobject

suchasthewelcome()functioninthepreviousexample.WhatNodedidforyou

intheabovecodewastocreatetheexportsobjectwhenitingestedyourfile.

Node has code for the require() function that sets everything up to be

exposed. Node takes your code from your file, passed in as a parameter, and

doessomethinganalogoustothefollowing:

functionrequire(file){

module.exports={};

...Thefileparameterisusedandthatfileisopened

...andcodeisparsedandtakenandplacedbelow.

//Yourextractedcode

module.exports.welcome=function(name){

console.log(“Hi”+name);

}

//Endofyourextractedcode

returnmodule.exports;

}

Anemptyobjectiscreatedinthefirstlineofthefunction.Thatobjectthen

has properties added to it, such as the function you see. Finally, the object is

returnedsoothercodecancallthisfunctionofftheobject.

Forthecodetousethefunctioninthemodule,itneedstousetherequire()

function.Therequire()functiontakesthenameofthefilethathasyourmodule

codeinit.Youreferencemymodule.jsasfollows,andcallthefunctionyouhave

exposed.

//fileserver.jsthatusesthesamplemodule

varw=require(“./mymodule.js”);

w.welcome(“Bob”);

Nodeactuallyhasaninternalmoduleobjectthatitcreatesforeachmodule

exposed.Thereisalotmorethatisgoingonbehindthescenesthanthat,butyou

don’tneedtoreallyknowanymoreofthedetails.Youcancertainlylearnmore

of the internals if you are curious by reading through the actual source code,

sinceitisanopen-sourceproject.

Amorecomplicatedmodule

You can hang multiple properties on the exports object, such as objects,

strings,numbers,arrays,etc.Inoneoftheprevioussamples,Ishowedyouthe

useoftheHTTPmodule.IthasacreateServer()functionattachedtoit.Thisisa

designpatternknownasthefactory designpattern.Youdon’tusethefunction

directly,aswasdonewiththewelcome()functioninthepreviousexample,but

callittogetanobjectthatyoucanthenuse.

Youcanalsoexposeaclassthroughaconstructorfunctionthatclientstake

andconstructthemselves,oryoucangofurtherandprovideafunctionthatdoes

the creation for them like the factory pattern I just mentioned. If you have

multipleclassestoexpose,thenyouwouldwanttousethefactorypattern.Ifyou

onlyhaveoneclass,thenyoucanexposethatwithaconstructorfunction.

Ifyouwandtoprovideaconstructorfunctioninamodule,itwouldlookas

follows:

//mymodule.js

vara=require('http');

functionWelcome(nameIn){

this.name=nameIn;

}

Welcome.prototype={

this.name=null,

showName:function(){

console.log(“Hi”+name);

updateName:function(nameIn){

this.name=nameIn;

}

};

module.exports=Welcome;

Thefollowingexampleshowshowthismodulecanbeused:

varWelcome=require('./mymodule.js');

varw=newWelcome(“John”);

w.updateName(“blah”);

w.showName();

The welcome() function acts as the constructor in this example. The

prototype keyword in JavaScript allows you to set properties that exist for all

instancesandarethusnotcreatedagainforeveryinstance.Modulesaresingle

instance cached anyway. As Node keeps references to each that are used and

givesthesameinstancebackeverytimeitisrequired.

NotehowIincludedtheusageoftheHTTPmoduletobeusedbythesample

moduleabove.Thisshowshowmodulescanrequireothermodulesfortheirown

functionality and do so with the standard require() function. These included

modulesarenotvisibleoutsideofthatmodulecode.Thecodeusingthemodule

can’tactuallygetaccesstotheHTTPmodule,unlessitalsohadarequire('http')

statementaswell.

ImportinsteadofrequirewithNodeversion10

TheJavaScriptlanguageitselfhasreleases,andisanimplementationbased

on a standard called ECMAScript. Node has an implementation of JavaScript

thatisonapathtoadoptmoreECMAScriptsyntaxasthatevolves.IntheES6

release of ECMAScript, a new module system was introduced that uses the

‘import’keyword.Version10ofNodetakesasteptowardsimplementingthis,so

you would no longer need to use ‘require’ statements if you prefer to use the

‘import’ keyword instead. People writing HTML browser applications are

already familiar with this new standard. Here is what the old way looks like

comparedwiththenewerway:

//OlderCommonJSsyntaxthatexistedinNode

constsm=require(“./somemodule”)

//ESMsyntaxforgettingthedefaultexport

importsmfrom“./somemodule”

8.5UsefulNodeModules

Ifyougotohttps://nodejs.org/api/,youwillfindtheofficialdocumentation

onthecoreNode.jsmodules.Glancethroughwhatistheresoyoucankeepitin

mindifyouneedtoreferenceitinthefuture.

Besides the modules that come with Node, there are plenty of other

installable packages from NPM. https://www.npmjs.com/browse/star is a site

thatliststhemostpopularones.

Keepinmindthatmanyof theNPMdownloadsareforcode modulesyou

useinsideanapplicationandothersaretoolsthatyoudownloadandrun.Some

code modules are also for use as express middleware. Some are for other

frameworkssuchasReactorAngular.

Here is a very small list of some code modules you might find useful in

Node.js code that you write. A few come with Node itself and the rest you

downloadfromNPM.

Module: Purpose:

async

Usedtoforceyourcodetoruninaworkflow.Insteadof

doingthingslikenestingcallbacks,youcanuseasynctoset

upsequentialcalls.Alsoused torunparallel functionswith

theabilitytoknowwhenallhavefinished.

child_process Forchildprocessspawningandmanagement.

cluster

ForsettingupaclusterofNode.jsprocessestodistribute

theloadacross.

events

Aprimitivemodulethatmanyothersarebuilton,toemit

orlistentoevents.

express

Web server functionality for configuring routes, serving

up static files and providing template data binding

functionality.

fs StandardOSfilesystemfunctionality.

helmet

For mitigating different types of HTTP security

vulnerabilities.

http

Thismoduleservesadualpurpose.Youcanuseittoset

upalisteningserviceforincomingHTTPrequests.Youcan

alsouseittomakeoutgoingHTTPrequests.

joi

Forperforming validation onHTTP request JSON body

properties.

lodash

A collection of very useful helper functions (see

https://lodash.com/docs/)

mongodb UsedtointeractwithaMongoDBdatabase.

net

Standard low-level networking functionality for servers

andclients.

os BasicutilityfunctionsforaccessingOSinformation.

process

The standard type of functionality for working with

processesonanoperatingsystem.

request SimplifiesmakingHTTP/Sclientcalls.

response-

time

DisplaystheresponsetimeforHTTPrequests.

socket.io

A higher-level way to build bi-directional

communicationsbetweenclientsandservers.

stream

A primitive module that many others are built on to

providereadableandwritablestreamsontopofdata.

url UtilityfunctionsforworkingwithURLs.

util

InternalutilityfunctionsthatNodeitselftakesadvantage

ofthatareexposedforyoutouse.

zlib Forcompressionanddecompression.

Chapter9:Express

The E in the MERN acronym stands for Express. Express is a module

commonlyusedinaNode.jsapplication.ItishowevernotapartoftheNode.js

installation but can be downloaded from NPM and installed separately and

integratedintoyourapplication.Expressisverypopularandconsistentlyoneof

the most downloaded Node.js modules on NPM. The Express module allows

youtoimplementthefunctionalityofawebserver.Itprovidesawaytospecify

theroutehandlingforincomingrequestsandsimplifiesresponsegeneration.

OneofthethingsExpressdoesforyourprojectistoremovetheneedforthe

HTTP module that comes with Node. The HTTP module that is built into

Node.js requires you to write a lot of code in order to set up the routing and

responses.Expressmakesallofthateasier.

ThefollowingarethekeycapabilitiesofExpress:

SpecifytheroutehandlingofincomingHTTPrequests.

Mechanism to inject middleware into a request to modify it as it

getspassedalong.

Easy integration of third-party middleware to provide extended

capabilitiesforrequestprocessing.

Provides a requestobject with properties and methods to look at

everythingconnectedwiththeincomingrequest.

Provides a response object to use to set everything up for a

response.

ConfigurationforJSONpayloadserving.

Configurationtoserveupstaticfiles.

Pairsupwithserver-sidetemplateenginestosetcontextandreturn

HTMLwithdatabinding.

9.1TheExpressBasics

As with any external Node module, you have to do an NPN install and

requireExpressinyourcode.Youcandothisatthetopofafilesuchasyour

server.jsfile.YoucanthenmaketheExpresslisteneractive.Hereissomesimple

codetosetupanduseExpress:

varexpress=require(‘express’);

varapp=express();

app.listen(3000);

I will walk you through the construction of the NewsWatcher sample

applicationandshowyouhowtoseteverythingup.Beforethathappens,Iwill

coverthebasicsofhowExpressisused.

Expressconfigurationsettings

As part of using the Express module, you will need to configure some

settings in your code. These determine how it works and are settings you just

need at startup time. You use the Express object set()function to do this. The

app.set(name,value)syntaxisusedforsettingavalueforpredefinedvaluesthat

Express uses as configuration. The set()method will configure values such as

whatporttolistenon.Hereisanexample:

varexpress=require(‘express’);

varapp=express();

app.set(‘port’,3000);

To disable a setting, you can call app.set(name, false). You use the

app.get(name) method to retrieve any set value. There are enable and disable

functions, but they are the same thing as calling set with a true or false. The

followingsettingscanbeusedforconfiguringthemodeofoperationofExpress:

SettingswithValueoptions:

case

sensitive

routing

A Boolean to determine route interpretation as to case

sensitivity. For example, if set to false, then “/News” and

“/news”aretreatedthesame.

env

A string value that sets the environment mode such as

“development”.Thisispurelyforyourusetosetandread.For

example, you would have logic to determine which URL

endpoints, database connections, etc. depending on if you are

running the code to try out new development code, or if the

codeisrunninginproduction.

etag

UsedtosettheETagresponseheaderforallresponses.Set

ittostrong,weak,orfalseifyouwanttodisableit.Youcanalso

passinacustomfunction.Thedefaultvalueisweak.

jsonp

callbackname

AstringtospecifythedefaultJSONPcallbacknamesuchas

?callback=, which is the default. JSONP to bypass the cross-

domainpoliciesinwebbrowsers.Notnecessarilyneeded.

json

replacer

A string to specify the JSON replacer callback. This is a

functionyoudefinetodecideonaproperty-by-propertybasisif

theyarereturnedonaJSONrouteresponse.

json

spaces

Specifies the number of spaces to use for JSON indenting

forreadability.

port Theporttolistenon.

query

parser

Youcan setthis tosimpleorextended.simpleis based on

thequeryparserfromNodeandextendedisimplementedbythe

qsmoduleandisthedefault.

strict

routing

ABooleantosettotrueifyouwantrouteslike“/News”and

“/News/”treatedasdifferentpaths.Thedefaultsettingisfalse.

subdomain

offset

A number that defaults to 2 for how many dot-separated

partstoremovetoaccessthesubdomain.

trustproxy

Tobeusedifyouhaveafront-facingproxy.Youwillneed

tosetthisuptobetrusted.Thisisfalsebydefault.

views Thedirectory(s)tousefortemplatelookups.

view

cache

ABooleanthatenablestemplatecaching.Iftheenvsetting

issettoproduction,thenthisvaluedefaultstotrue,otherwise,it

defaultstofalse.

view

engine

A string to specify the template engine to use with your

providedtemplates.

powered-by

A Boolean(defaults is true)to enable the"X-Powered-By:

Express"HTTPheadertobereturned.Youwouldsetittofalse

soasnottoreturnittopreventanyhackersfromknowingtoo

muchaboutyourimplementation.

TheExpressobjecthasapropertynamedlocalsonwhichyoucanplaceyour

own custom properties that you might want to associate with your Express

application.Youusetheapp.localsobjectasshowninthefollowingexample:

app.locals.emailForOps='Help@myapp.com';

Listening

Withcodeinplacetomanageyoursettings,allyouneedtodonextismakea

functioncalltoallowtheNode.jsruntimetorunitsplatform-specificcodeand

setupasocketconnectionforthegivenportontheservermachinetolistenon.

AnyincomingconnectionstotheIPaddressofthemachine,atthatspecifiedport

numberwillbeboundthroughtoyourcodetorespondto.Hereisthecodetodo

that:

varserver=app.listen(3000,function(){

debug('Expressserverlisteningonport'+server.address().port);

});

Withsome understanding ofthe initialization andsettings, you canlook at

implementingafullyfunctionalwebservicewithHTTPrequestroutehandling.

9.2ExpressRequestRouting

WhenanHTTPrequestcomesintoNode,therequestwillmakeitswayto

theExpress code you have writtento service it. YourNode.js instance willbe

runningonamachinethatishostedandexposedontheInternet.Nodewillbe

executinginaprocessonthatserver,andthroughtheExpresscodeitwouldbe

listeningonaportsocketforincomingconnections.

HTTPrequestscan cometo yourserverto rendera browserpageor fulfill

RESTwebserviceAPIrequeststhatdealwithJSONpayloads.AURLrequest

wouldbelikethefollowing:

http://newswatcherscale-env.us-west-2.elasticbeanstalk.com/news?region=USA

If you use query strings as shown above, you can get those values un the

Expressrequesthandlers.ThefollowingdiagramillustratesExpressrouting:

Figure37-Expressrouting

Ifyoureallyneedto,youcansetupthecodetoservicelowerlevelTCPor

UDPtypes ofconnections.Youcan researchtheNode ModulesNet andUDP.

ThischapterwillonlybeconcernedwithservicingHTTPrequestswithExpress.

Routes

OnceyouhavetheExpressJavaScriptobjectthroughtherequirestatement,

you can use methods that set up the servicing of HTTP requests. Express will

thenlistenforconnectionsonthespecifiedportanditcanunderstandthevarious

HTTPverbsandbreakdownwhat isbeingpassedinaspart oftheURLpath,

querystring,andtheHTTPheadersandbody.

TheExpressobjectgivesyoufunctionstouseforhandlingthevariousHTTP

verbs(get,putetc.).HerearethestandardverbsthatwouldbeusedforCRUD

typeoperationstoexposeanAPIinyourservicelayer.

app.get(path,callback);//Readitem(s)

app.post(...);//Createanewitem

app.put(...);//Replaceanitem

app.patch(...);//Updateanitem

app.delete(...);//Deleteanitem

functioncallback(req,res){

res.send(“Sendsomethingback”);

};

ExpresssupportsthefollowingHTTPmethods:

checkout

copy

delete

get

head

lock

merge

mkactivity

mkcol

move

m-search

notify

options

patch

post

purge

put

report

trace

unlock

unsubscribe

Please see Express’s documentation for the complete list of supported

methods.Thegeneralsignaturelooksasfollows:

app.<METHOD>(path,callback[,callback...])

<METHOD>wouldbereplacedbyoneofthemethodssuchasget.Thefirst

parameteristheURLpath.ThisisnotthecompleteURL,buttheportionofit

afterthedomainname.

The second parameter is the callback function that gets called for that

request. You can actually provide multiple callbacks and they will each get

calledsequentially.Iwillcovermoreonthislater.

Routepathscanbestrings,stringpatterns,orregularexpressions.Theycan

alsobeanarraythatcombinesanyofthementionedformats.Asastring,aroute

pathcan be things like“/books”. Be aware thatthe query string portionis not

consideredpartofthepath.

Youcanuse“/”tospecifythatallpathsshouldbepickedup.Or,ifyouomit

thepathparametercompletely,allpathswillbepickedup.

Thecallbackfunctionyouprovidehasatleasttwoparametersinitsstandard

form.Thefirstparameteristherequestobject,whichcontainsinformationabout

theincomingrequest.Thesecondparameteristheresponseobject,andisused

to send back a response, such as serving up an HTML file or sending back a

JSONpayload.

ThefollowingexampleshowstheuseofapureRESTstyleURL.Youjust

needtoprovidetheroutingpaththatoccursafterthedomainportionoftheURL.

HereistheURLandthewayyouwouldspecifythepath.

//http://mysite.com/news/categories/sports

app.get(“/news/categories/sports”,callback);

For each verb, such as get, you might have different routes for different

resourcesthatarebeingretrieved.Hereissomecodethatsetsupmultiplepath

routesforthegetHTTPverb,eachreturningsomethingunique:

app.get(‘/about’,function(req,res){

res.send(“Aboutpage”);

});

app.get(‘/news’,function(req,res){

res.send(“Newspage”);

});

Be aware that the ordering of your route handling code is very important.

Any incoming request is basically consumed by the first path that is found to

handleit.

Asinglepathformultipleverbs

Ifyoufindyouhaveseveralverbsthatallrespondtothesamepath,youcan

specifythemtogetherbyusingtheroute()method.Thismighthelpyoualleviate

typinginthepathmultipletimes.Anexampleofthisis:

app.route('/customer')

.get(function(req,res){res.send('Getacustomer');})

.post(function(req,res){res.send('Addacustomer');})

.put(function(req,res){res.send('Updateacustomer');});

Allverbsatonce

Besidesthestandardverbs,therearealsothemethodsallorusetorespondto

allverbsoftheincomingrequests.Thefirstroutehandlerbelowisforallverbs

that are for the path /test. Then the second is set up for everything else and

returnsa404-NotFoundcode.Theusefunctionhereisnotusingapath,soitis

forallverbsandallpathsnotservicedyet.

app.all('/test',function(req,res){

resp.type('text/plain');

resp.send('Thisisatest.');

});

app.use(function(req,res){

res.type('text/plain');

res.status(404);

res.send('404-Notfound');

});

Advancedpathspecification

Sofar,youhaveseenthesimplestofcaseswithroutehandlingpaths.There

is a technique that can give you more advanced parsing capability for a URL

path.BelowisoneoftheURLsfromapreviousexample:

//http://mysite.com/news/categories/sports

app.get(“/news/categories/sports”,callback);

Theproblemwiththisexampleisthatyoumightneedtoservice20different

categories of news stories. For example, what about the paths

/news/categories/science or /news/categories/politics? It could get very

monotonoustosetuproutesforeachandeveryone.Tomakethiseasier,Express

allowsyoutosetupplaceholderparametersinthepaththatyoucanthengetat

laterinthecallbackcodeandthenhavethecodehandleit.

To be able to retrieve parts of a path, you use a special syntax in the path

parameter by placing a colon character in the string. This then sets up a

JavaScriptpropertyyoucanlateraccessinthehandlingfunctionaspartofthe

request object. The code below sets the path up with a colon. A property will

thenbeavailableontherequestobjectasshown.

//http://mysite.com/news/categories/sports

app.get(“/news/categories/:category',function(req,res){

console.log('Yourcategorywas'+req.params.category');

});

Withtheabovecode,youjusthavetheoneroutethatcanserviceallrequests

for news stories and can just feed that category into the backend retrieval

mechanism.

Thereq.paramspropertycanbeusedasanarray.InJavaScript,thatmeans

you can also access it as shown next. The following example shows this, and

alsoshowsthatyoucanhavemorethanoneoftheseparameterssetuptouse:

app.get('/products/:category/:id',function(req,res){

console.log(req.params[0]+req.params[1]);

});

YoucanalsoconstructyourURLssothattheycontainquerystrings.Fora

querystringthatyouwanttoprocesssuchas/news?category=sports,youdon’t

needanyspecialsyntaxinthepathparameter.Justspecifythepathuptothestart

ofthequerystringandstopthere.Thename-valuepairsofthequerystringwill

automatically be parsed out and made available to you as part of the query

property of the request object. For this example, there would be a property

namedcategory,withavalueofsports:

//http://mysite.com/news?category=sports

app.get('/news',function(req,res){

console.log(req.query.category);

});

9.3ExpressMiddleware

Atthispoint,youknowhowtosetupcallbackfunctionsthatgetexecuted

foragivenincomingrequest.NowI’llintroducetheconceptofmiddlewareasa

means of inserting route processing code that will happen before your ending

routehandler runs. Some middleware code will be provided by Express, other

middlewarecanbe by modulesdownloaded fromNPM, andother middleware

canfromwhatyouwrite.

Theconceptofmiddlewareisthatyoucanhavecoderunasaninsertedstep

thatisplacedbeforeyouractualroutecallbackgetsrun.Middlewarecodegets

chained together in a series of calls that you specify. Express Middleware can

thenactuponandmodifytherequestobjectthatisbeingpassedalongtheway.

Youcandothistoreusecodeacrossmultipleroutes.

Some inserted middleware terminates the request, as it takes care of

everything and the requests never even go to any of your route handling. An

example of this is the Express static file serving middleware. You might also

havesomemiddlewarethatinterceptscallstoverifyauser’sauthorizationand

doesnotcontinueiftherequestisdeterminedtobeinvalid.

Anotherexampleofmiddlewaremightbesomethingthatcachescontentfor

you.Anotherexamplewouldbeafunctionthatlogsalloperations.

Manyavailablemiddlewaremodulesaresimpletoadd,yetverypowerfulin

whattheyprovide.I’llshowyouseveraloftheminthischapter.Conceptually,

you can take the Express routing diagram shown previously and modify it as

follows to show middleware being injected that passes functionality down the

chain:

Figure38-Expressroutingwithmiddleware

Middlewarefunctionslookalmostidenticaltowhatyouhaveseenalreadyas

callbacks. They are just callback functions with the same signature you have

seenandthushaveaccesstotherequestandresponseobjects.Thismeansthat

therequestobjectcanbemodifiedbeforebeingpassedalong.Forexample,the

request body in the response object could be modified to have additional data

addedtoitbeforeitgetstoitsfinaldestination-handlingcallback.

You can read the Express documentation to learn about available modules

youcandownloadfromtheNPMrepository.Middlewarecanaccomplishthings

like authentication, caching, logging, session state, cookies, etc. These act as

sharedpiecesofcodethatyoucanuseacrossallorjustcertainroutes.

MiddlewareextendsthecapabilitiesofNode.jsbeyonditscorefunctionality.

Forexample,let’ssayyouarewritingawebserverthatwillserveupstaticfiles,

suchasimagefiles.Node.jsallowsyoutodothatifyouwritethecodetodoso.

However, there is a module that acts as middleware in Express that makes it

incrediblyeasytoimplement.Themodulethatdoesthisisthestaticmoduleand

itcomesaspartoftheExpressinstall.Itcanbehookeduptoserveupstaticfiles

withverylittlecode.

Note: Don’t confuse the concept of a middleware module with the general

concept of a module in Node.js. They are still provided through the requires

function,butareuseddifferentlythanregularNodemodules.

Hookingupyourowncustommiddleware

Normally,routeprocessingstopsatthefirstmatchthatisfoundforaURL

pathandthenacallbackisrun.However,ifyoumakeoneminormodificationto

yourcode,youcanstringtogethermultiplecallbacksforthesameroute.Notice

theonemodificationmadebelow:

//Firsthandlerfortheroute

app.get('/test',function(req,res,next){

...

console.log(“Gotherefirst”);

next();

});

//Secondhandlerforthesameroute

app.get('/test',function(req,res){

...

console.log(“Gotheresecond”);

});

Thedifferenceisthatthefirstroutecallbackfunctionhasathirdparameter

namednext. This parameter is a function, and its usage tells Express that you

wantthecallbacktoactasmiddlewarecodetobeinjectedbeforetheactualend

routeiscalled.Theorderisimportantasstatedbefore.

next()isafunctionthatyoucallwhenyourmiddlewarecodeisdonewithall

processing.Youmustcallnext()ortherequestwillbeabandonedandnotmake

ittoyourendhandler.Intheexample,thefirsthandlerrunsandthen,becauseof

thenext()functioncall,executioncontinuestothesecondhandler.

Thepreviousexamplecodecouldalsobestructuredsothatthecallbacksare

notseparatedout.Youthenjustlistoutcallbacksoneafteranother.Youwould

needtohavethecodeinthecb1()functionthatcallsnext()orcb2()willnotbe

run.

app.get('/test',cb1,cb2);

functioncb1(req,res,next){

console.log(“Gotherefirst”);

next();

});

functioncb2(req,res){

console.log(“Gotheresecond”)

});

Universalmiddleware

You can hook up middleware that will get inserted into every single route

and for every single verb. To do this, you simply use app.use(). This sets up

Expresstousethisfunctionacrossallincomingrequests.Youcanleaveoffthe

pathinthiscase,as“/”isthedefaultpathifyoudon’tprovideone.Ofcourse,

you may want to provide a path so that the middleware only gets run for a

certainpath.

app.use('/',function(req,resnext){

...

next();

});

Youcaninsertasmuchmiddlewareasyouneedforyourroutes.Theorderin

whichyoulisttheminyourcodewillbetheorderinwhichtheyaresequenced

through.Beawarethatcertainthird-partymiddlewarefromNPMarerequiredto

be placed before others. Refer to the middleware’s documentation for more

information.

I will now show you a practicalexample of some custom middleware you

mightwanttoimplement.Let’ssay thatyouhaveaspecialpath thatyouonly

want administrators to have access to. You can create a function that does

validationbeforeallowingtherequesttoproceedforfurtherprocessing.Hereis

howyoudothat:

//Middlewareinjection

app.all('/admin/*',doAuthentication);

app.get('/admin/stats',returnStats);

app.get('/admin/approval',approval);

With the app.all(), the authentication happens for all verbs and acts as

middleware. Inside the doAuthentication() would be code to determine the

authenticityoftherequest.Ifitwasdetectedtobeinvalid,thenyouwouldnot

callnext()andtheothertworoutehandlerswouldneverbecalled.Youwillsee

somethingsimilarbeingdoneintheNewsWatcherapplication.

If you call next() and pass an error object parameter, then that route

terminates from being handled normally. Error handling middleware then gets

invoked.Thistypeoferrorhandlingwillsoonbeexplained.

Parametermiddleware

Express allows you to set up a middleware callback function for a given

parameter property you defined in other route handlers. You do this with the

app.param()function.Thiscallbackiscalledbeforeanyroutehandler.Hereisan

example:

//Usingtheglobalparamhandler

app.param('id',function(req,res,next,value){

console.log(“someonequeriedid“+value);

if(value!=99)

next();

});

app.get('/products/:category/:id',function(req,res){

console.log(req.params[0]+req.params[1]);

});

Ifyoudon’thaveanyrouteswith“id”inthem,thentheparam()callbackwill

nevergetcalled.Theparamcallbackwillbecalledbeforeanyroutehandlerin

which the parameter occurs. You still need to call next() to continue the

processing.

Routerobject

If you really have a lot of routes and want to subdivide them for better

organization,youcanusetherouterobjectandkeepeachintheirownmodules.

Inthatway,youcanisolateyourlogicandnothaveitaffecttheotherroutesyou

havesetup.

You set the router objects up independently. Until you activate them with

app.use(),theywillnotbefunctional.Thefollowingisanexampleofsettingup

some middleware and some routes in for two different routers and then

activatingthemintheapp:

varrouterA=express.Router();

varrouterB=express.Router();

//Setuproutesthatwillendupwithnews

routerA.get('/weather',function(req,res){...

});

routerA.get('/sports',function(req,res){...

});

//Setuproutesthatwillendupforblog

routerB.get('/tech',function(req,res){...

});

routerB.get('/art',function(req,res){...

});

//Setsup/news/*

app.use('/news',routerA);

//Setsup/blog/*

app.use('/blog',routerB);

Middlewareerrorhandling

Anymiddlewarefunctioncanreturnanerror.Itdoessobycallingnext(err),

whereerris an Errorobject. Execution ofthe middleware and anysubsequent

routing is ended and processing of the error takes place. To process the error,

Expresshasadefaultfunctionthatitcallsthatwritestheerrorbacktotheclient.

Youhave the option of providing your own function or chain of functions for

handlingmiddlewareerrors.Ifyouprovideafunctionorchainoffunctions,then

thedefaultonewillnotbecalled.Inthefollowingexample,notehowthereare

fourparametersontheerrorhandlingmiddlewarefunction:

app.get('/test/:id',function(req,res,next){

if(req.params.id==0)

next(newError('NotFound'));

next();

};

//Amiddlewareerrorhandlingfunction.errisanerrorobject.

app.use(function(err,req,res,next){

console.error(err);

res.status(500).send('Somethingbadhappened');

});

Onceexecutionhasshiftedtotheerrorhandlingmiddleware,youcaneven

returnbacktoregularrouteprocessingifyoudoanext(‘route’)call.Doingthat

willjumpyoutowhateveristhenextdefinedroutehandler.

The diagram of Express routing can be further expanded to add in

middlewareerrorhandling.

Figure39-Expressroutingwithroutehandingandmiddlewareerrorhandling

Usingnext()evenifyourhandlerisnotmiddleware

I should clear one thing up for you though. Just because you add a next

parameter on a route handler does not mean you are implementing some

middleware.Forexample,youwillmostlikelyneednext()asaparameteronall

yourendroutehandlersinordertodocentralerrorprocessing.

As an example, the following code is an end route being handled by the

Expressrouter object. As it iswritten, you assume nothing willgo wrong and

justcarryoutsomeoperation:

varrouter=express.Router();

router.delete('/:id',function(req,res){

res.status(200).json({msg:'Loggedout'});

});

Whatifyouwantedtodetectanerrorandusemiddlewareerrorhandlingas

explainedintheprevioussection?Thisiswhereyouwillneedtoaddthenext

parameter to be called if there is an error. If there is no error, the route will

complete by calling the res.status()function to send back a response. Nothing

elsegetsintheway,andifrunningnormally,youdon’tcallnext()becausethere

isnothingelsetochain.

In order to use the error handling middleware, an end route itself must be

abletopasscontroltotheerrorhandlingchain.Todothat,thecodeneedstobe

modifiedtoaddinthenext()functiontobecalled.

Asmentioned,theendingroutehandlernevercallsnext()withoutgivingit

anerrorparameter.Thefollowingexampleshowstheadditionalerrorhandling.

Theerrorhandlercouldthensendaresponsewithanerrorcodeandmessage.

varrouter=express.Router();

router.delete('/:id',authHelper.checkAuth,function(req,res,next){

if(req.params.id!='77')

returnnext(newError('Invalidrequest'));

res.status(200).json({msg:'Loggedout'});

});

Staticfileservingmiddleware

One middleware module that comes with Express is the static middleware

thatallowsinterceptionofrequestsforfilesandreturnsthem.Thisalleviatesthe

needforyoutoprovideanendroutehandlerofyourown.Forexample,ifyou

wanttoserveupjpgimagefiles,youcanusethefollowingcode:

varexpress=require(‘express’);

varapp=express;

//Middlewareinjection

app.use('/images',express.static(‘images’));

app.listen(3000);

app.useissettingupthemiddlewareroute.Insteadofthecallbackfunction

beingprovidedbyyou,youinsertthecallforusingexpress.static().Thistakes

careofeverythingforyoutosendaresponseback.InyourNode.jsprojectcode,

youwouldneedtoprovidethefolderofimages.AnHTMLpagecouldaccessan

imageasfollows:

<imgsrc=”http://yousite.com/images/someimage.jpg”/>

Asasecondparametertoexpress.static(),youcanpassinanoptionsobject

that can have the following properties on it: dotfiles, etag, extensions,

fallthrough,immutable,index,lastModified,maxAge,redirect,setHeaders.

Forexample,setHeadersisafunctionyouwouldusetosetheaderstosend

with the files. Another example would be setting the lastModified property to

trueandtheLast-Modifiedheadervalueissettothedateofthefilebeingsent.

Formoreinformation,refertoExpress’sdocumentation.

Third-partymiddleware

There are a growing number of third-party modules that provide Express

middleware for your applications. Go to the Express site

(http://expressjs.com/resources/middleware.html) to find a list of modules you

candownload.

To use the middleware, you typically call app.use(<middleware>). This

meansthatallpathsandverbswillflowthroughit.Mostthird-partymiddleware

that intercepts routes before you get them will call the next() function so that

processingwilleventuallyreachyourcode,ifyouhavethatneed.

HereareafewusefulExpressmiddlewarecomponentsforyourreference:

passport

Usedtoauthenticaterequests.Youcansetthisuptologa

person in using OAuth (i.e. through Facebook), or federated

availablethroughthepassportmodule.

body-parser

Thismiddlewareinterceptsany HTTPPostverb requests

that have body data, such as from a form submit. The

middlewarecoderunsandthenbythetimeyourhandlercode

runs, the response object has a body property with sub-

propertiesoffofitforeachofthebodyvalues.

varbodyParser=require(‘body-parser’)

app.use(bodyParser());

//Inyourhandler,youcanlookatthevalues

app.post(‘/’,function(req,res){

console.log(req.body);

}

Youcan specifythat JSONis tobe parsedand placedin

thebody.Querystringvaluescanalsobeplacedintothebody

objectforyou.

varbodyParser=require(‘body-parser’)

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({extended:true}));

This will compress requests that pass through the

middleware.Youwouldplacethisstatementbeforeanyother

middlewareorroutes,unlessyouonlywantedcertainroutesto

be compressed. If you look at the headers of a returned

response, you would see the headers have an entry for

compression Content-Encoding.

varcompression=require('compression')

varexpress=require('express')

varapp=express()

app.use(compression())

cookie-

parser

This middleware does all the work to make available a

cookiepropertyonyourrequestobject.

varcookieParser=require(‘cookie-parser’)

app.use(cookieParser());

//lookatallofthepropertiesonbody

app.post(‘/’,function(req,res){

console.log(req.cookies);

}

errorhandler

Thisaccomplishesthesendingbackofstacktracestothe

clientwhenanerroroccurs.Makesuretoonlyusethiswhen

runninginadevelopmentenvironment.

varerrorhandler=require(‘errorhandler’)

app.use(errorhandler());

express-

session

Server-sidesessiondatastorage.

express-

simple-cdn

UsageofCDNfor staticassetservingwith multiplehost

support.

helmet

Helpful for mitigating several HTTP security

vulnerabilities.

response-

time

Response time tracking to add the X-Response-Time

header. The value inserted is in milliseconds. You could use

this to track your SLA over time and be alerted as to what

needsfurtherinvestigationforperformanceoptimization.

morgan

Thisisforrequestlogging.Thisfreesyouupfromwriting

anyofyourownconsole.logstatements.Forexample,incoming

HTTPrequestsgothroughthismiddlewareanditslogsthose

requests to the console window. You can also specify the

formatofthelogginganddirecttheoutputtoafile.

varfs=require('fs')

varmorgan=require('morgan')

//createawritestream(inappendmode)

varaccessLogStream=fs.createWriteStream(__dirname+'/access.log',

{flags:'a'})

app.use(morgan('combined',{stream:accessLogStream}))

multer Multi-partformdata,oruploadingfilesinchunks.

serve-

favicon

Forcustomizingtheiconinthebrowser.

timeout ForsettingatimeoutperiodforHTTPrequests.

express-

validator

Forvalidationofincomingdata.

connect-

redis

SessionstoreusingRediscache.

connect-

timeout

For routes that might run into some backend processing

issuesandneedtobelimitedintheamountoftimetheytake,

youcanusethistocutthemoffandreturnanerror.Youstill

needto determinewhat theright approachis for termination

andresubmissionofrequests.

varcto=require('connect-timeout')

app.get(‘/some_questionable_route’,cto(‘5s’),

function(req,res,next){

...somepossiblylongrunningcode...

...checkreq.timeouttoseeifitisevertrueand

...thenreturnfalse

returnnext();//finishedprocessingintime,gotonextfunction

function(req,res,next){

res.send(‘ok’);

}

);

vhost

For routing by hostname different sub-domains. i.e.

www.mysite.comversusapi.mysite.com.

express-

stormpath

Userstorage,authentication,authorization,SSO,anddata

security.WillworkwiththeOktaAPI.

9.4ExpressRequestObject

Let’slookmorein-depthattheusageoftherequestobjectinExpressroute

functionhandlers.Therequestobjectcontainsalltheinformationyouneedfor

digestingtheincomingrequest.Forexample,youhaveseenhowtheproperties

req.params and req.query are used. You have also seen some third-party

middlewarethataddsmorepropertiestotherequestobject.Hereisareferenceto

some of the properties that are available. You can refer to the Express

documentationtofindthecompletelist.

The request object is a parameter of your express route callback function.

Youcannameitanythingyoulike.“req”isagoodnameforit.Thefollowing

exampleshowshowtogetthecompleteURLthatthisrequestcameinfrom.

app.get(‘/’,function(req,res){

console.log(req.url);

});

Hereisareferencetothepropertiesthatareavailableontherequestobject.

RefertotheExpressdocumentationforthecompletelist.

Requestobjectproperties:

app

Areferencetotheinstanceoftheexpressapplicationobject.

params

This is used to access route parameters. You need to first

havesettheroutespecificationstringandthenyoucanusethe

paramspropertyoftherequestobject.

app.get('/user/:id',function(req,res){

res.send('user'+req.params.id);

});

query

Used to get the URL querystring. Aproperty will exist on

thequeryobjectforeach.

//ForURL“/users/search?q=Smith”,

app.get('/users/search',function(req,res){

console.log(req.query.q);

});

body

Containspropertiesofkey-valuepairsofdatasubmittedin

therequestbody.Youneedtoaddthebody-parsermiddleware

forittowork.

varbodyParser=require('body-parser');

app.use(bodyParser.json());

app.post('/',function(req,res){

console.log(req.body);})

route

Thisisanobjectthathaspropertiesofthecurrentroutesuch

aspath,keys,regexp,andparams.

Whenusing the cookie-parser middleware,this property is

anobjectthatcontainscookiessentbytherequest.Eachcookie

thatisattachedisapropertyonthecookiesobject.

req.cookies.someName

signedCookies

Exists if the cookies have been signed and protected from

tampering.

req.signedCookies.someName

TheremoteIPaddressoftheincomingrequest.

protocol

Suchashttp,https,ortrustedifsetupwithatrustedproxy.

secure

ThishasavalueoftrueifSSLisineffect.

headers

TheHTTPheadersyoucanaccess.

req.headers['x-auth']

url

TheURLoftherequest.

path

Thepathpartoftherequest,withoutthequerystring.

route

Anobjectthatcontainsthematchedrouteandlotsofother

propertiessuchasthemethodandfunctionthathandledit.

hostname

ThehostfromtheHTTPheader.i.e.example.com

subdomains

The subdomain part that is in front of the hostname. i.e.

[“blah”,stuff”]iffromstuff.blah.example.com.

xhr

Thisissettotrueiftherequestcamefromaclientcallsuch

asfromXMLHttpRequest,whichhadsettheX-Requested-With

field.

Hereisareferencetosomeofthemethodsthatareavailableontherequest

object.RefertoExpress’sdocumentationforthecompletelist.

Requestobjectmethods:

get(field)

Togettherequestheaderfields.

req.get(‘content-type’);//i.e.“text/plain”

To check if a certain type is available, based on the

Acceptheaderfield. If whatyou sendin asa parameter

accepts(types)

doesnot match one of thevalues, then you will receive

anundefinedreturn.

req.accepts(‘html’);

is(type)

Tofindoutwhattypetheincomingrequestis.

req.is(‘text/html’);//i.e.returnstrue

acceptsLanguages(lang

[,...])

BasedontheAccept-Languagefieldoftheheader,it

returnsthefirstlanguageonamatch,orfalseifnoneare

accepted. Similar calls are acceptsCharsets and

acceptsEncodings.

varlang=req.acceptsLanguages('fr','es','en');

if(lang){

console.log('Thefirstacceptedis:'+lang);

}else{

console.log('Noneaccepted');

}

9.5ExpressResponseObject

Requestsareroutedtoyourcallbackbecauseofaroutingpathyouhaveset

up. Youwill eventually return a response back to the requester. The Response

objectiswhatyouusetodothatwith.YoushouldatleastsendbackanHTTP

status code. You might also return some HTML, text, or better yet, a JSON

payload.

Methodsontheresponseobjectarecombinedandhaveacumulativeeffect

onthereturnedresponse.Theexamplecodebelowsetsthestatusof200OKfor

asuccessfulHTTPrequest:

varexpress=require('express');

varapp=express();

app.get('/',function(req,res){

res.status(200);

res.set({‘Content-Type’:‘text/html’});

res.send(‘<html><body>Somebodytext</body></html>’);

});

app.get('/test_json',function(req,res){

res.status(200);

res.set(‘jsonspaces’,4);

res.json({name:’me’,age:37});

});

app.listen(3000);

Inthefirstroute,theContent-Typeissetas“text/html”andthenthesend()

methodisusedtofinishthereturnedresponsewithsomereturnedHTML.The

secondroutereturnssomeJSON.

Theresponseobjecthasmanyusefulpropertiesandmethods.Eachusageof

theresponseobjectisusedinsideafunctioncallback.Youcannameitanything

youlike.“res”isagoodnameforit.

Here is a reference to the properties and methods that are available on the

responseobject.RefertotheExpressdocumentationforthecompletelist.

Responseobjectproperties:

App

AreferencetotheExpressapplication.

headersSent

ABooleanvaluethatistrueifHTTPheadershavebeensent.

locals

Local variables scoped to the request, might be identical to

app.locals.Atemplatecanusetheseforitsdatabinding.

Responseobjectmethods:

status(code)

Usedtosetthestatusofareturninthecaseofanerror.

res.status(404);

accepts(types)

Forcontentnegotiationonareturn.

if(req.accepts('text/html')=='text/html')

res.send('<p>Hello</p>');

}elseif(req.accepts('application/json')=='application/json')

res.send({message:'Hello'});

}else{

res.status(406).send('NotAcceptable');

}

set(field[,

value])

Forsettingthefieldsoftheresponseheader.

res.set({‘contentType’:‘text/plain’,‘ETag’:‘123’});

get(field)

Retrieveswhatthesettingisforaheaderfield.

res.get(‘contentType’);

Thepathtoredirecttoinsteadoftheoneitcameinat.You

redirect([code,]

URL)

canprovideanoptionalstatuscode.A302“Found”isthe

defaultvalueofthecode.Youcanalsoredirectrelativetothe

currentURLoftheservice.

res.rediect(‘http://example.com’);

cookie(name,

value,[options])

Setsacookie.Thenameistheidentifierofthecookie.

Thevalueparametercanbeastringorobjectconvertedto

JSON.Theoptionsparametercansetupthingslikedomain,

expires,httpOnly,maxAge,path,secure,andsigned.

res.cookie('rememberthis','1',{maxAge:900000,secure:true});

json([body])

SendaJSONbodyback.

res.json({msg:'Hello'})

jsonp([body])

SendingJSONwithJSONPsupport.

res.jsonp({msg:'Hello'})

attachment([path

tofile])

Youcansetanattachmenttobereturned.Ifyoupassa

parameter,itisexpectedtobeafile.TheContent-Disposition

andContent-Typearesetforyou.

res.attachment('path/to/logo.png');

//Content-Disposition:attachment;filename="logo.png"

//Content-Type:image/png

sendFile(path[,

options][,fn])

Transfersafile.

res.sendFile(‘me.png’,{maxAge:1,root:’/views/’},function(err){});

end([data]

[,encoding]

Usethistoendtheresponsewithoutanydatabeing

returned.

res.status(404);

res.end();

format(object)

Usethisifyouaregoingtoreceiverequestsforcontentof

morethanonetype.Youcanlineupmultiplepiecesofcode

foreachAcceptHTTPheadertype.

res.format({

'text/plain':function(){

res.send('Hi');

'text/html':function(){

res.send('<p>Hi</p>');

'application/json':function(){

res.send({message:'Hi'});

'default':function(){

res.status(406).send('NotAcceptable');

}

});

append(field[,

value])

Addsthespecifiedtextandvaluetotheheader.

res.append('Warning','199Miscellaneouswarning');

send([body])

SendingofanHTTPresponse.

res.send({message:'Hello'});

sendStatus(code)

Setstheresponsecodeforthereturn.

res.sendStatus(200);

render(name,[,

data][,callback])

Templateresponsesending.Seethenextsectionofthis

bookformoreinformation.

res.render('user',{name:'Tobi'},function(err,html){

...

});

9.6TemplateResponseSending

OneofthethingsthatExpressenables,issendingserver-sideHTMLthathas

beengeneratedfromtemplatesthathavedataboundtothem.Thereareseveral

populartemplatelanguagesthataresimilartoHTMLmarkupthataresupported

throughExpress.Iwillhighlightjustone,butyoucaninvestigateothers.

Using the Express response object, you can formulate a response to send

backwithafunctionnamedres.render().Thisfunctiontakesafilethatcontains

thetemplateasoneparameter,referredtoasthe view.Asasecondparameter,

youcanprovidethedataobjectthatbindstothetemplate.Thetemplatethatyou

haveloadedthroughExpressbindsthedataandproducestheresultingHTMLas

theoutputtopassbackontherequest.

You can pass a third parameter as a callback function to get the rendered

string and process any errors that might have occurred. Here is what the code

lookslikethatutilizesatemplatetosendbackasaresponsetoarequest:

app.set('views',path.join(__dirname,'views'));

app.set('viewengine','jade');

app.get('/test',function(req,res){

res.render('test.jade',{

title:'MyNewsStories',

stories:items

});

First,youneedtohaveincorporatedtheNPMjademoduleintoyourproject.

ThenyouneedtomakecallstotellExpresswhatdirectorythetemplatefilesare

inandwhattemplateengineyouareusing.Expresswilltheninternallyusethe

Jademoduleyouhaveincludedinyourproject.

Noticethetwoapp.set()callsusedtoconfiguretheuseofJade.Theapp.get()

setsuptherequestroutewithahandler.Itisinthathandlerfunctionyouhave

thecalltorenderthetemplatewiththegivendatatobindtoit.

RefertoJade’sdocumentationto learnaboutthe templatesyntax. Itusesa

curlybracesyntaxtobindtoproperties.HereisaJadetemplatethatcouldtake

thepassedindatacontextandbindthosevaluestoelements:

//test.jadefilecontentforthetemplateview

doctypehtml

html

head

titlemyjadetemplate

body

h1Hello#{title}

div.newstbl

eachstoryinstories

div.storyrow

a(href=story.link)

img.story-img(src=story.imgUrl)

h6#{story.title}

Beawarethat,ifyouusetemplaterendering,youarerelyingonserver-side

renderingofyourHTML.IfyouprefertoutilizeaSPAarchitectureforaclient-

side browser application, you would not want to do this. In part three I will

describehowtoreturnHTMLthatusesReactcomponentDOMrenderingforthe

client-side SPA application. You can also do server-side rendering (SSR) with

React,eitherto returnall yourrendered pages, orjust afew ofthem. A React

Nativeapplicationwillalsobediscussed.

Chapter10:TheMongoDBModule

Thischapterisoneofthemostimportantonesinparttwoofthisbook.That

isbecauseoneofthemainpurposesofaservicelayeristoprovideaccesstothe

data layer. To do that, you will be utilizing a Node.js module that has been

createdtointeractwithMongoDBonthebackend.Youwillbelearninghowto

usethe“mongodb”modulefromtheNPMrepository.

Itisnecessarytofirstincludethismoduleinyourpackage.jsonfilesothatit

ismadeavailableinyourproject.Theusualrequire()statementisthenusedto

make functionality available in your code. I will cover that again when you

constructtheNewsWatchersampleapplication.

Note:ThemongodbNPMmoduleAPIisquitemassiveanditwouldtakea

large book to document it all. This book cannot make you an expert in all its

usage. For example, there are functions to create and delete collections and

perform other administrative duties that I have chosen to perform through the

AtlasmanagementportalandCompassapp.Youshouldcertainlymakeaquick

passthroughtheAPItoseewhatothercapabilitiesithasthatyoumightwantto

takeadvantageof.LookforthemongodbmoduleontheNPMsiteandfromthere

findalinktothedocumentation.

TheMongoClientobject

To begin with, your code needs to establish a connection to a MongoDB

server.Todothis,youusetheconnect()functionofthemongodbMongoClient

object. After establishing a connection, there are a lot of useful functions for

interacting with a MongoDB collection. The function signature for connect()

looksasfollows:

connect(urlConnectionString,[options],[callback])

The first parameter of the function is the URL of the service endpoint for

your MongoDB instance. The second parameter contains options that can be

usedforsettingsontheserver,suchasforareplicaset,etc.Thelastparameteris

yourcallbackfunction,whereyouwillreceivetheclientobjectthatthenallows

youtobeconnectedtoadatabaseandacollection.

IfyouhaveanincorrectURL,youwillgetanerrorreturned.Youcanfind

theURLconnectionstringtousebyopeningtheAtlasmanagementportal.Click

theCONNECT button for the cluster. Click Connect Your Application. You

willseetheconnectionstringlisted.YoucanclicktheCOPYbuttontocapture

it.URL-encodeyourusernameandpasswordiftheycontainanycharactersthat

arenotreadilyusedinaURLwithoutconvertingthem,suchas‘#’or‘@’.

Figure40-Atlasdatabasepagewithconnectionurl

Inmorereadabletext,thisissomethinglike:

mongodb+srv://test:pwd@cluster0-gas8f.mongodb.net/test?retryWrites=true

There are placeholders there for a password. You should also go to the

Securitytababovetheclusterandcreateauserloginthatyouwilluseinyour

code. This is different from the account login you use with the administrative

Atlasportal.Youneedtochooseasettingtogivetheuseraccountreadandwrite

accesswhenyoucreatetheuser.

Hereissomeexamplecodecallingconnect()toestablishaconnectionusing

theURLfromtheaboveexample.

vardb={};

varMongoClient=require('mongodb').MongoClient;

MongoClient.connect("<yourconnectionstring>",function(err,client){

db.collection=client.db('newswatcherdb').collection('newswatcher');

});

The code in the callback uses the database connection and calls the

collection()functiontogetthe“newswatcher”collectionobjectthatcanthenbe

usedtoperformCRUDoperations.

I can now show you a few of the methods exposed with the mongodb

module. The focus will be on learning the functions necessary to perform the

CRUDoperations.

10.1BasicCRUDOperations

With the collection object now obtained, you are ready to learn about the

CRUD operations. You can now learn about the four fundamental CRUD

operations necessary to utilize a MongoDB database. In the terminology of

MongoDB,forsingledocumentinteractions,CRUDtranslatestothefollowing

functions:

insertOne()

findOne()

findOneAndUpdate()

findOneAndDelete()

I’llnowwalkyouthrougheachofthefourfunctionsandshowyouhowto

usethem.Later,youwillputtogethertheNewsWatchersampleapplicationand

use them all again, plus a few others. At that time, you will make your code

morerobustwitherrorhandling.

Note:ThereareactuallyvariationsoftheCRUDfunctionslistedabove.For

example, to find multiple documents, it is necessary to use the find()function.

Whenreadinganyofthedocumentation,becarefultopayattentiontoanytext

statingthatitisdeprecated.

Create

The following is the function signature used for creating a document in a

collection:

insertOne(doc,[options],[callback])->{Promise}

Thefirstparameter istheJavaScript objectthat youwantto haveinserted.

This then gets created as a BSON document. If you do not provide the _id

property in your passed in JavaScript object, MongoDB will generate one for

youwhenitstoresthedocument.

Yourcallback function will have as the first parameter an error object that

youcanchecktoseeifsomethingwentwrongoncreation.Ifyoudon’tprovide

a callback function, a Promise is returned for you to use. The following code

passesanobject,andusesthecallbackfunction.

//Thisexampledoesnotusetheoptionsobjectparameter

db.collection.insertOne({property1:“Hi”,property2:77},function(err,result){

if(err)console.LogError(“Createerrorhappened”);

elseconsole.log(JSON.stringify(result.ops[0],null,4));

});

TheoptionsparameteroftheinsertOne()functionisanobjectthathasaset

ofpropertiesonit.ThissameobjectisusedformanyofthecallsintheAPI.I

willdescribeithereforyourreference,asyouwillseethisusedagain.Allthe

propertiesofthisobjectareoptional.Hereisatablethatdescribestheproperties

oftheoptionsobject:

RequestOptionsObjectProperties:

Property: Purpose:

The write concern (how the acknowledgment

works).

wtimeout

Thetimeoutyouwantforthewriteconcern.

Tospecifythejournalwriteconcern.

serializeFunctions

Booltoserializefunctionsonanyobject.

forceServerObjectId

Boolforserverassignmentof_idvaluesinsteadof

driver.

bypassDocumentValidation

To bypass schema validationin MongoDB 3.2 or

higher.

session

Optionalsessiontousefortheoperation

Theresultparameterinthecallbackfunctionhassomepropertiesyoumight

be interested in. One of them is the ops property. It was used in the previous

example and contained the returned document. The result properties are as

follows:

Property: Purpose:

insertedCount

Thenumberofdocumentsinserted.

ops

Anarrayofallthedocumentsinserted.

insertedId

ThegeneratedObjectId.

connection

Theconnectionobjectused.

result

ThecommandresultobjectreturnedfromMongoDB

Read

Toretrieveasingledocumentfromacollection,usethefindOne()function.

Thefollowingisthefunctionsignatureforretrievingadocument:

findOne(query,[options],[callback])->{Promise}

Thefirstparameteristhequerycriteria.Youcangobacktochapterthreeto

reviewwhatthatlookslike.Whatthisfunctiondoesistosimplyreturnthevery

firstdocumentthatmatchesthequerycriteriaandnomore.Thecallbackhasan

errorobjectfollowedbythedocumentreturned.Hereisanexample(notusing

theoptionsparameter)offindOne():

db.collection.findOne({email:"nb@abc.com"},function(err,doc){

if(err)console.LogError(err);

elseconsole.log(doc);

});

Optionsisanoptionalobjectwith20optionalpropertiesyoucanuse.Refer

tothemongodb module’s documentationfor acomplete listofproperties. The

followingisadescriptionofafew(therearemanymore)ofthem:

Name Type Description

fields

object

The projection criteria to specify the

fieldstoincludeorexclude.

hint

object Totellthequerywhatindexestouse.

explain

boolean

Return an object with query analysis

andnottheresult.

raw

boolean ReturntheBSON.

readPreference

ReadPreference

|string

Which machine in replica set to read

from,suchastheprimaryorsecondary.

maxTimeMS

Number

How long to wait in milliseconds

beforeabortingthequery.

session

ClientSession

Optional session to use for the

operation

findOne()returnsaJavaScriptpromiseifnocallbackwasprovided.

Youcanusetheexplainoptiontolookathowwellyourindexesareworking.

You will see some JSON returned that gives you useful data about the query.

Makesuretoremovethisoptionafterwardasitpreventsyouractualresultfrom

beingreturned.

Update

The following is the function signature for updating documents in

MongoDB:

findOneAndUpdate(filter,update,[options],[callback])->{Promise}

Thefirstparameteristhefilterparameterwhichisthequerycriterianeeded

toidentifythedocument.Thesecondparameterisfortheupdateoperatorstobe

used.Yourcallbackfunctionwillhaveasthefirstparameter,anerrorobjectthat

youcanchecktoseeifsomethingwentwrongontheupdate.Hereisanexample

usagethatusesanoptiontohavetheupdateddocumentreturnedinthecallback:

db.collection.findOneAndUpdate(

{email:"nb@abc.com"},

{$set:{name:"Charles"}},

{returnOriginal:false},

function(err,result){

if(err)console.log(err);

elseif(result.ok!=1)console.log(result);

elseconsole.log(result.value);

});

Thefollowingisadescriptionoftheoptionsobjectproperties:

Name Type Description

projection

object

Theprojectioncriteriatospecifythefields

toincludeorexcludeinthereturn.

sort

object

Species a sorting order for multiple

documentsthatarematched.

maxTimeMS

Number

How long to wait in milliseconds before

abortingthequery.

upsert

boolean Createthedocumentifitdidnotexist.

returnOriginal

boolean

Set this to false if you want the updated

documentreturned

Session

ClientSession Optionalsessiontousefortheoperation

findOneAndUpdate()returnsapromiseifnocallbackwasprovided.

Delete

The following is the function signature for deleting a document in

MongoDB:

findOneAndDelete(filter,[options],[callback])->{Promise}

Thefirstparameterisforthequerycriterianeededtoidentifythedocument.

Yourcallbackfunctionwillhave,asthefirstparameter,anerrorobjectthatyou

can check to see if something went wrong on deletion. Here is an example

usage:

db.collection.findOneAndDelete({email:"nb@abc.com"},function(err,result){

if(err)console.log(err);

elseif(result.ok!=1)console.log(result);

elseconsole.log("UserDeleted");

});

Hereisadescriptionoftheoptionsobjectproperties:

Name Type Description

projection

object

Theprojectioncriteriatospecifythefields

toincludeorexcludeinthereturn.

sort

object

Species a sorting order for multiple

documentsmatched.

maxTimeMS

Number

How long to wait (milliseconds) before

abortingthequery.

Session

ClientSession Optionalsessiontousefortheoperation

findOneAndDelete()returnsapromiseifnocallbackwasprovided.

10.2AggregationFunctionality

InthedatalayerchapterswheretheMongoDBcapabilitiesforqueryingwere

covered, I omitted one specialized type of query. What I omitted was the

capabilityofMongoDBtoperformaggregationoverthedata.Aggregationgives

youtheabilitytoreportonsummarizationsofdatasuchasgrouping,orfinding

thesum,min,ormaxacrossavalue.

This gets rather involved, so I left it until now to even mention this

capability.Youreallyneedtomakethisafocusofsomeseriousstudyinorderto

masterallthatispossiblewiththeaggregationcapability.

Hereisasimpleexampletogiveyouafeelforhowitworks.Imaginethat

for the example bookstore used in prior examples, you wanted to find out the

number of customers living in each state. You would use the aggregate()

function for this. The following example is the signature of the aggregate()

function:

aggregate(pipeline,[options],callback)->{null|AggregationCursor}

The pipeline parameter is an array of MongoDB supported aggregate

commands.Thinkoftheseasstagesthedataisbeingpipedthroughfromoneto

thenext.Therearequiteafewaggregateoperatorsyoucanstringtogether.

Hereisanexamplethatwouldgivethecountofpeoplebystate.Noticehow

wechainthefunctiontogotothetoArray()function.

db.collection.aggregate(

[

{$group:{"_id":"$address.state","count":{$sum:1}}}

]).toArray(function(err,result){

console.log(result);

});

Theresultmightbeasfollows:

[{_id:'UT',count:54},

{_id:'KS',count:988},

{_id:'FL',count:1259}]

Thecallbackhasanerrorobjectthatyoucancheck.

There are operators like $match and $project that you can insert to help

narrowdownthedocumentsandwhatpropertiesarepassedthroughthepipeline.

10.3WhatAboutanODM/ORM?

ThisbookshowshowtoconnectdirectlytoMongoDBwithanodemodule

createdspecificallyforthatpurpose.ButthereisanotherNPMmoduleyoucan

use that approaches interfacing with MongoDB in a completely different way.

Thatwouldbewiththe“mongoose”module.

Thosefromarelationaldatabase backgroundwillunderstandthatthere are

such things as Object Relational Mappings (ORMs) for connecting to a SQL

Server.Perhapsyou arefamiliarwithEntityFrameworkfor .Net,orHibernate

forJava?Theequivalentinadocument-baseddatabaseiscalledanObjectData

Mapping(ODM).

Lookup“mongoose”onNPMorGitHubandyouwillfindanODMthatsits

ontopofMongoDB.Thismodulecanbeusedinsteadofthemongodbonethat

wecoveredinthisbook.

The mongoose module adds an additional abstraction. With this, you get

featureslikeschematizationofthedataandvalidation.

Here is what code looks like that uses the mongoose module to save and

queryadocument.Iwilltakethesampleofthecustomerdocumentthatmight

exist in the online bookstore example. I will cut back on the number of

propertiesthough,soitisashorterexample.Hereiswhatsomecodewouldlook

likethatusesmongoose:

varmongoose=require('mongoose');

mongoose.connect('<TheusualconnectionURL>');

varCustomer=mongoose.model('Customer',{name:String,age:Number,email:String});

varc=newCustomer({name:'Aaron',age:32,email:'ab@blah.com'});

c.save(function(err){

if(err)console.log(err);

});

mongoose.model('Customer').find(function(err,customers){

console.log(customers);

});

There are ways to accomplish each of the ODM capabilities on your own.

For example, to add in a simple module that helps you do server-side input

validation,youcanuse“express-validator”or“joi”.Butwhygotoallthework,

ifanODMalreadyexists?AnODMmodulemaybethewaytogotogiveyou

morerobustnesswithyourapplication.

10.4ConcurrencyProblems

Onceyourapplicationstartstohavemultipleconcurrentusers,youmayrun

intoissuesupdatingdocumentsinMongoDB.Let’stakeanexamplewhereyou

haveasingledocumentthatcontainsalistofhighscoresforanonlinegameand

youwanttokeepthetopfivescoresacrossallplayers.Asusersfinishagame,a

callismade tosubmit thescore theyachieved andinsert itinto thelist oftop

scores. High scores are best, so if a new score is posted that should be added

becauseitishigherthanotherentries,thelowestscoreisdroppedoffthelist.

If multiple players all submit their scores at the same time, you can

understand that some contention might arise with updates to this single

document.Hereisadiagramofhowthismightlook:

Figure41-Topscorecontentionillustration

All of these users are causing score insertion code to run in parallel. Each

request gets sent to a Node.js Rest API endpoint route to allow multiple

simultaneous calls to read and update the single document in the MongoDB

DBMS.

Whatwouldhappeniftwoscoresaresubmittedforprocessingattheexact

sametime?Eachrequestwouldfirstreadthedocumentandtheninsertascoreif

itishigherthanthelowestnumberinthelist.Thedocumentwouldthenbesent

back for replacement in the collection. Here is a sequence diagram that

illustratestheproblemwithtimeflowingtoptobottom:

Figure42-Simultaneousupdate

If you look carefully, you can see that the final document is not what it

shouldbe.YouwouldwanttoseebothMegandJiminsertedatthetopandthen

KateandJoedroppedofffromthebottomofthearray.Instead,Megisatthetop

andonlyJoedroppedoff.Jimisreallygoingtobedisappointedwhenhechecks

thehighscoresanddoesnotseehisnamelisted.

Neither of the updates knew the other one was happening and so the last

write“wins”.ThisisaclassicproblemandnotsomethinguniquetoMongoDB.

There have indeed been many solutions invented over the years to solve the

problem.Forexample,somedatabase technologiesofferlocking.The problem

with this is that locking can be tricky in your own code and requires you to

implement detection of stale locks. MongoDB does not support API level

locking. It does, however, do this internally on its own for certain calls you

make.

ThesolutionthatMongoDBanditsunderlyingstorageengine(WiredTiger)

implement for you is termed optimistic concurrency. If you use the

findOneAndUpdate() function, this will happen at the document level. If you

were, however, to use read, followed by update functions, simultaneous reads

willstillgetthesamedataandonewritewillprevail.

With findOneAndUpdate(), if two calls come in, the first one pauses the

second one from doing anything until the first call has finished on that

document. The second call will not even do the read until the first one has

completeditswriteofthedata.Thesecondcallkeepsretryingforatimeuntilit

issuccessful.

Let’s go back to the initial sequence diagram and draw this out one more

timewithretriesputin:

Figure43-Simultaneouswriteswithoptimisticconcurrency

Optimistic concurrency retry logic can be inefficient if your database is

alwaysbeinglockedandretriesareconstantlyhappening.Imagineiffourusers

consistentlyupdatetheirscoresatthesametimeeverysecondoverandover.If

fourattemptsaremadeatthesametime,onlyonecansucceedandthentheother

threemusttryagain.Thesecondroundhasthreeattemptsandoneworks,andso

forth.

ThepointisthatyoumightbewastingMongoDBcomputetime.Thismay

causeyour MongoDB service to perform poorly. Youcan always look atyour

design and see if you really need to have one single document that is always

being contended for. You may run into cases where you simply have a “hot”

document with lots of simultaneous updates all of the time, as was illustrated

withthegamescores.

ImaginemultipleindependentNode.jsprocessesthatareallhittingasingle

backendMongoDBdatabase.Thismeansyouwillhavetocomeupwithaway

tocoordinateacrossindependentprocesses.

Ifyouwanttoreallyimplementtheultimatearchitecturetohandlemassive

scalingandavoidtheconflictsyougetwithoptimisticconcurrency,youneedto

createasinglequeuethatisoutsideofallyournodeprocesses.

Youwouldplaceyourupdateoperationrequestsinanexternalqueue,such

asanAWSSQSresourceforglobalaccessacrossallnodeprocesses.Youthen

need to create a single unique node process that would watch the AWS queue

andprocesstherequests.Thefollowingexampleshowsthissolution.Thedots

next to the Scaled Web API represents multiple machines with load balancing

acrossnodeprocesses:

Figure44-Queueserializationsolution

Thiswouldalsobethesafestsolutionifyoureallyhadtoensureconsistency

andrepeatability.Thereisonemaindrawbackwiththissolution—youneedto

deal with the issue of returning to the initial caller if the operation succeeded.

Thismightnotbeimportant,butitcouldbe.

You can decide to take the “fire-and-forget” approach and always return

success. Perhaps, losing a top score for some reason can be considered

acceptableiflatertheworkerrolefailstoinsertascoreforsomereason.

Whatif,however,youhadabanktransactionthatrequiresanotificationone

wayortheotherbacktotheuser?Youwouldneedtoimplementsomewaytodo

an asynchronous return. There are notification mechanisms that are available

that would solve this. It is possible you could have some callback from the

queuingsystemthatcouldreturnifthetimewasinthemillisecondrangetonot

delaytoomuchtheHTTPrequestreturn.YoucoulduseWebSocketsorHTTP2

capabilitiestopushbacktothebrowsertheresult.Therequestwouldbequeued,

andanidreturnedthattheUIcanusetopollwithtogetthecompletionstatus.

Chapter11:AdvancedNodeConcepts

Therearesomeparticularlydifficultissuestobeawareofthatrequirecareful

coordinationacrosstheNodeecosystem.Therearecertainintricaciesthatmust

be handled properly. Done incorrectly, it can be disastrous. Done correctly,

everythingwillhum along.Thischapter willcovera fewof thesubtletiesyou

mightneedtoaddress.

11.1HowtoScheduleCodetoRun

Atimerfunctioncanbeusedtoschedulecodetorunatalatertime.Thiscan

be done as a one-time request, or it can be set up to happen on a recurring

interval.Youcanalsolookintousingthecronmoduletoscheduletherunningof

periodic Node code. The following code schedules a function to run in five

secondsandshowshowtopassinaparameter:

setTimeout(myFcn,5000,“five”);

functionmyFcn(param1){

console.log(“Hi%s”,param1);

}

Thecallbackisnotgoingtohappenexactlyat5000milliseconds,butNode

willdoitsbesttofititinwhenitistimeandthecallbackisthengiventoV8to

run.

YoucanusethefunctionsetInterval()toschedulearecurringfunction.Ifyou

want to cancel the recurring timer, you can cancel it at any time with

clearInterval().Thefollowingexampleshowshowthiswouldlook:

varid=setInterval(myFcn,5000,“five”);

...

clearInterval(id);

Be aware that Node will run your function over and over, even if the

previouscallhasbeenblockedandhasnotyetcompleted.Ifyouwantacallback

executed only if the previous one has completed, you could write your own

implementationofsetInterval()topreventthatbehavior.

Another function named setImmediate() is available that does not have a

timer go off, but places the callback to be executed after I/O, but before

setTimeout()andsetInterval()eventsareprocessed.

Ifyouareinterestedinschedulingcodetorunatanevenhigherpriority,you

canusetheprocessobjectnextTick()function.Becarefultonotusethisunless

you are using it responsibly. This call will place your callback to be executed

above all other processing in the event queue, even before I/O callbacks are

executed.Ifyoudothistoooften,atsomepointyouwillcompletelycutoutall

I/Oprocessing.

11.2HowtoBeRESTful

ItisuptoyoutodesigntheURLendpointsthatyourNodeapplicationwill

respond to. You can create a RESTful service and even support OData or

GraphQL if you like. Youare also free to include support for query strings, if

thatisyourpreference. Ifyou gothe RESTfulroute, thereare afew thingsto

keepinmind.

ThefirstthingtorememberisthatRESTfulwebservicesareintendedtobe

stateless.Ifyouarescalingyourserver-sideNodeprocessandusingthecluster

module, or have scaling through Elastic Beanstalk, then you most likely want

stateless servers. Any of your servers can process any incoming client call. If

you really want state information kept around, you can store state information

with the client, or have it cached for server-side retrieval in something like a

Rediscache.

Make sure to carefully design your REST API URLs so that they make

sense.Therearestandardthingstoconsidersuchasusingnounsandnotverbsin

yourpaths.Youmayrunintosomedilemmas,butthereisprobablyananswer

foryourchallengeinaforumsomewhere.

When you come out with a new version of your REST API, you need to

makethatapparentandperhapssupportmultipleversionsforawhile.Youcan

strive to keep your API as backward compatible as possible. You can insert a

versionnumberintoyourpathtomoveclientsfromoneversiontoanother.For

example, if you had “/api/users” as a path, you could have clients start using

“/api/v2/users”fornewfunctionality.Youcanalwaystackaquerystringonthe

endtospecifytheversioningsuchas“?Version=2015-12-22”.AnHTTPheader

settingisalsopossiblesuchas“x-version:2015-12-22”.

11.3HowtoSecureAccess

You should conduct a review of all your data connection points and

scrutinize all data that is being transferred and stored. Make sure to take

appropriate precautions with sensitive data you are safeguarding for your

customers.Informationsuchasahomeaddresscanbeusedtoidentifyaperson

and should never be leaked. Financial and medical records many times have

lawsandregulationsconcerningtheirstorageandtransmission.

You are not just trying to ensure your business interests are safe, you are

responsibletosafeguardyourcustomersfromanyharm.

One very important detail you need to work out is how users will identify

themselves and be allowed to access your Web API from a client application.

The other security concern you need to solve is how to prevent any

eavesdropping or man-in-the-middle type of hacks as information flows back

andforthfromtheclienttotheWebAPIservice.Thissectionwillexplorethese

andotherrelatedtopicsandpresentasolutionforeach.

Accesstoken

Once a person is identified by their logging in, a Web Service needs to

recognizethemandallowthemaccesstodataassociatedwiththeiraccount.One

possiblemeansofuserinteractionauthorizationistohavethemsigninandthen

useasessioncookiewitheveryrequestcomingin.

Another similar mechanism is to generate an access token and have that

passed in with every client request. There is a standard way to do this with

somethingcalled aJSON Web Token(JWT). A JWTis something thatcan be

generated on the server-side in response to a client login request when they

present a username and password. The JWT is basically an encoded set of

informationabouttheuser thatissignedtomake sureitis nottamperedwith.

HereisthesequencediagramforhowaJWTiscreatedandpassedfromlayerto

layer:

Figure45-JWTtokenpassing

You can pack into the JWT whatever you want. Put in information that

wouldhelpyouinyourprocessinggoingbackandforth.Onenicethingwould

betoaddintheIPaddressaswellastheHTTPuser-agentheadervalue.Thiscan

thenbevalidatedonlatercallstomakesureyoustillhavethesamepersonusing

thetoken.Youcanalsoset anexpirationtimeon thetokenso thataperson is

requiredtologineverysooften.

Tooffloaduserauthentication,youcanusetheOath2standard.Thisallows

youtohave aperson redirectedtosome otherwebpresence theyalready trust

and log in there first. Users might prefer this, as they only need to have one

singlesign-oncredentialtomanage.

WiththeNPMpassportmodule,youcanimplementOath2todelegatethe

userauthenticationtoanexternallytrustedsitesuchasGoogleorFacebook.A

tokenofauthenticityispassedbackthathasinformationaboutwhotheyare.

Note: The JWT should always be transferred using HTTPS because it can

easily be decrypted. Make sure it doesn’t contain anything that could

compromiseyoursecurity.Thetokenisintheheadersuchas“x-auth:<token>”

or“Authorization:Bearer<token>”ifyouwanttomimicstandardslikeOIDC.

Alldatatrafficshouldbeencrypted

Oneof the first things todo,once you have sufficientmomentumon your

code,is toimplement certificate-basedauthenticationand encryption usingthe

HTTPSstandard.Thiswillenableyoursitetobeviewedaslegitimateandalso

ensurethatdataistransferredusingtheencryptedTLS/SSLprotocol.

Ifyousearchontheinternet,youwillfindcodesamplesthatshowyouhow

to configure Express to require HTTPS. If your Node.js service were to be

hosted on a machine directly exposed on the Internet, this is what you would

needtodo:

consthttps=require('https');

constfs=require('fs');

constoptions={

key:fs.readFileSync('keys/agent-key.pem'),

cert:fs.readFileSync('keys/agent-cert.pem')

};

https.createServer(options,(req,res)=>{

res.writeHead(200);

res.end('helloworld\n');

}).listen(8000);

However, if you choose to use a PaaS solution, the above code is not

necessary.ThisisbecauseyourNode.jsserviceishiddenbehindtheserverthat

acts as the reverse proxy and load balancer. This means that the AWS Elastic

Beanstalk service external-facing load balancer needs to be configured for

HTTPS. You will see how this is set up when the sample application is put

together.

11.4HowtoMitigateAttacks

WhenyouexposeaserviceontheInternet,itwillbevulnerabletoattacksof

all kinds. Some attacks might be intentionally malicious and others just

annoying.Allthreatsshouldbetakenseriously.Ataminimum,theycandisrupt

your service, which is unacceptable. Beyond that, attacks can steal sensitive

informationanddodamagetoyourcustomersandtoyourownreputation.

Obviously,asimplenativemobilephonegameoftic-tac-toewouldnothave

as large an attack surface as a three-tier e-commerce application. The more

infrastructure and code that you have, the larger your attack surface will be.

Hackerswilllookforthevulnerabilitythatiseasiesttoexploit.Yoursecurityis

onlyasgoodasyourweakestpointofattack.

Toreallygetanaccuratelookatallpossibleavenuesofattack,youneedto

draw out a data flow diagram that shows all the processes, interactions, data

stores,anddataflows.Eachprocesswouldrepresentthosethatyouown,orones

thatyouarerelyingon.Someofthosecouldbeclassifiedascompletelyexternal

andpossiblyoutofyourhands,buttheyshouldstillbeonthediagram.

Foreachoftheelementsinthediagram,youwouldwanttodosomeanalysis

todeterminewhatthreatscouldexist.Forexample,ifyouhadaSQLdatabasein

yourdiagram,youwoulddeterminewhatitisstoring,howdatagetsinandout

and what configuration and administration are happening. You might discover

thatyourSQLdatabasewouldbevulnerabletoaSQLinjectionattack.

With all of your analysis done, you would mitigate each of the

vulnerabilities. In some cases, it might simply involve a few lines of code, in

other cases, you might need to re-architect parts of your system. To really

addressthistopic,youshouldbuyabookspecificallydedicatedtothattopic.I

willnowpresentafewsecurityconcernsassociatedwithNode.jsandWebAPI

interactions.

NevertrustANYinput!

Onebasicstrategytorememberistonevertrustanyinput.Alwaysdowhat

youcan to validate alldata before using it.Look into using thenode modules

“joi” or “validator”. Using a data layer ORM/ODM can give you these data

validation capabilities, but you still need to sanitize your data from script

injections.Hereissomevalidationusingthejoimodule:

varschema={

displayName:joi.string().alphanum().min(3).max(50).required(),

email:joi.string().email().min(7).max(50).required(),

password:joi.string().regex(/^[a-zA-Z0-9]{3,30}$/)

};

joi.validate(req.body,schema,function(err,value){

if(err)

returnnext(err);

});

You can see that only alphanumeric characters are allowed for the user

displayname.Thejoimoduleisalsomakingsurenoextrapropertiesexistonthe

body.Thesetypesofrestrictionsareextremelyimportant,sodon’tunderestimate

theirusefulness.

Another issue is data transmission size. What if someone started sending

reallylargeJSONpackages,oronesthathadextraobjectsorpropertiesinthem?

Youcansetupyourbody-parsermiddlewaretoturndownrequeststhataretoo

large. The default size is 100kb so you can make that smaller just to be safe.

Hereishowyousetthatup:

app.use(bodyParser.json({limit:'10kb'}));

Let’s now looks at some of the types of attacks that could occur on your

exposed Web API. None of these are really specific to Node.js. They exist

becauseofthefundamentalwaythatbrowsersandHTTPwork.

DOS/DDOSattack

Thedenial-of-service(DoS)ordistributeddenial-of-service(DDoS)attackis

where traffic is thrown at your web app to try to bring it down. It might be

possibletooverwhelmitsothatothersarepreventedfromusingit.Ifsuccessful,

the attack will deny service to the actual people that are intended to use it. In

somecases,itmightactuallyleadtoincorrectbehaviorofyourapplication,soas

toexploititforothergains.

The distributed version of the attack just means that the attacker is

employing multiple distributed machines at once. The term bot is commonly

used, meaning that these machines are set up to run scripts or programs that

carry out the attack and constantly enlist other machines to also participate. It

actslikeavirusandreplicates.

ADoSattackispurelymaliciousandwouldrarelyhappenjustbyaccident.

Regardless,youneedtobepreparedforitandmitigatethisrisk.Youcanbesure

thatbige-commercesiteslikeWalmart.com,Amazon,andeBayseethesekinds

of attacks and take them seriously. An attack like this might even cause your

scalinginfrastructuretokickinunnecessarilyandstartcostingyoumoremoney

incloudoperatingcosts.

I must, of course, bring up what has already been mentioned – never do

anythingcomputeintensiveonthemainNode.jsthread.Thisisbecause,ifyou

havea lot of requests coming in that trigger someintensive synchronous code

thenyouwillhaveyourprocessbasicallyunabletorespond.Thiswilljustmake

iteasierforarealDoStooccurifyouletyourmainthreadgetoverloaded.Let’s

nowlookatwaystomitigateaDoSattack.

WhenyoucreateyourElasticBeanstalkNode.jsapplicationenvironmentin

the first place, it sets up Nginx to act as the reverse proxy and load balancer.

Nginx can be configured to limit the rate per IP address as well as limiting

connectioncountperIPaddress.

Another approach you can take using AWS is to set up an AWS API

Gatewayinfrontofyourservicelayer.Thatwillgiveyoualotofwhatyouneed

fordefense,suchasthrottlingperconnectiontoheadoffaDoSattack.Besides

that, you also get authorization, reporting, and API consumption of your web

APIcontractinacentralsharedway.

YoucouldalsodosometypeofIPblockingonyourown,suchastracking

the access time per IP and then limiting each IP to once per second access.

CheckouttheNPMmoduleexpress-rate-limit.Ofcourse,youcoulduseacloud-

mitigationproviderthatwouldusetheirexpertisetotrackpatternsofattackand

identifyDDoSattacksanddisablethem.

XSS-Cross-SiteScripting

AnXSShackiswheresomeforeignscriptgetsinjectedandrunas partof

yourweb-renderedsite.Alikelyvulnerabilitywouldbewhereyouareaccepting

input from the user and then later re-displaying that input back to them and

othersthatviewthesite.ThebrowserdoesnotbothertostopJavaScriptthatwas

maliciouslyputin,asitcannottellthedifference.Especially,ifyouconsiderthat

yourapplicationallowedtheusertoentersomethinginthefirstplace.Normal

userswillnotbetypinginmaliciousscriptstoberun,itisthehackersthatlove

todothisforfunandprofit.

Let’s take an example of something that could affect the NewsWatcher

application. In that application, people can comment on a shared news story.

This is an occasion where input from the user is accepted and later displayed

back to them. Let’s say that a malicious user enters the following text as a

commentonanewsstory,insteadofsomenicecomment:

Nowallotheruserslookingatthecommentwillbeaffected.IntheUIcode,

youmighthavesomeHTMLthatdisplaysallofthecommentsandtheDOMas

follows:

<p>'<script>alert("Hi");</script><imgsrc="smiley.gif">'</p>

</li></ul>

Everyonewillnowseeanalertboxandalsoacutelittlesmileyfacestaring

backatthem.Youneverintendedforthistohappen,butyoudidnothingtostop

it.Thesavvyhackercouldeven hacktheclient-sideJavaScriptcodeand mess

withtheJSONbeforeitgetssentbacktotheserver.WebAPIssimplycannever

trustthedatathatissenttothem

Imagine if the hacker referenced some script across the internet that really

wreaked havoc. If the hacker understood what your API was on the backend,

theycould runany commandas ifthey werea loggedin personand reallydo

somedamage.Thismeanstheywouldhavehijackedtheusersession.

Thisgoesbacktothesimplestatementthatyoushouldnevertrustuserinput.

Tomitigatethis,youcouldtakeactiontovalidatetheinputasyoucollectit.For

example,in the NewsWatcher code,you validate things likethe username and

don’tallow anythingbut alphanumericcharacters. Yoursanitizationcould also

scanallcharactersandchangeacharacterlike‘<’into“<”.

Fortunately,fortheNewsWatcherapplication,whenyouareusingReactand

renderyourdata,Reactdoestheworktodisableanyscriptsfromrunning.React

simplydisplaystheactualtextwithoutlettingthebrowserinterpretit.Inthecase

of the example above, you would actually see the literal string

“<script>alert("Hi");</script><img src="smiley.gif">” in the comment list and

allwouldbegood.

CSRF-Cross-SiteRequestForgery

ACSRFhackiswherearequestismadetoasiteyouarecurrentlylogged

intowithyourbrowser.Youwouldhavealreadybeenauthenticatedandhadan

authorization cookie stored by the browser. The attacker would trick you into

viewingapagetheyhadsetupandasthatwasloadedinthebrowser,itwould

runa script thatwould send a requestto the siteyou had already beenlogged

onto.

Asanexample,let’ssayyouwereloggedontoyourbankingwebsite.Now,

whilestillloggedon,youopenupanemailthatwasfromamaliciousattacker

thatsaid:“Clickhereandwinamilliondollars!”.Whenyouclickonthelink,

thedestinationURLdirectsyoutotheirmalicioussitethatloadsapagethatruns

ascriptthatsendsrequeststoyourbankingsiteandtransfersmoneytothemand

changesyourpasswordatthesametime.Sinceyouarealreadyloggedon,the

browserhappilysendstheauthenticationcookiealongwiththerequestandyou

arehacked.Thebankingsitehadnoideathatthisrequestwasnotvalid.

Our NewsWatcher sample application does not use cookies, so it is not

vulnerable to this attack. The JWTtoken sending is under the control of your

clientcodeandisnotautomaticallysentbythebrowserlikeacookieis.

If you do end up implementing some design that is vulnerable to a CSRF

hack,youcanusethe“csrf”NPMmoduletoimplementamitigation.Thiswill

createasecrettokenthatonlyyoursiteknowsthatisonlysentfromyourpages.

TheNPMHelmetmodule

Ihavediscussedeachofthesecurityconcernsanddiscussedmitigations.In

thissection,youwilltakealookattheHelmetNPMmodulethatwouldgiveyou

theabilitytofurthermitigatepossibleattacks.

The Helmet module tweaks your HTTP headers to set things up to utilize

certainbestpracticesforsecurityriskmitigations.TheHelmetmoduleworksas

Express middleware by injecting itself into the request-response chain. It does

notdoanythingthatyoucan’tdobyhand.Irecommenditthough,asitwould

takeyoualotmorelinesofcodeforyoutoaccomplisheverythingthatitdoes

withasinglelineofcode.

IwillshowyousomecodethatwillbethestartingpointwhenusingHelmet.

ThiscodewillsetupthingslikeenforcingHTTPS,mitigatingclickjackattacks,

certainXSSmitigations,andattacksbasedonMIME-typeoverridingattacks.

Youcantakethedefaultsandfurtherspecifyanydeviationsfromtherethat

youlike.RefertothedocumentationforanyofthespecificHTTPheadersyou

wanttoindividuallycontrolonyour own.Thefollowingcodeshowsyou how

easyitistousehelmet:

varexpress=require('express');

varhelmet=require('helmet');

varapp=express();

app.use(helmet());//Takethedefaultstostartwith

Theoneusageyoudoneedtocontrolonyourownisthatwhichisusedwith

thesettingofaContentSecurityPolicy(CSP).Thisbasicallyletsthebrowserbe

aware of where resources can come from. This will then prevent resources

unknowntoyoufrombeinginjected.ThebasicusageofhelmetwithCSPadded

becomesthefollowing:

varexpress=require('express');

varhelmet=require('helmet');

varcsp=require('helmet-csp');

varapp=express();

app.use(helmet());//Takethedefaultstostartwith

app.use(csp({

//Specifydirectivesforcontentsources

directives:{

defaultSrc:["'self'"],

scriptSrc:["'self'","'unsafe-inline'",'ajax.googleapis.com',

'maxcdn.bootstrapcdn.com'],

styleSrc:["'self'","'unsafe-inline'",'maxcdn.bootstrapcdn.com'],

fontSrc:["'self'",'maxcdn.bootstrapcdn.com'],

imgSrc:['*']

//reportUri:'/report-violation',

}

}));

Ifyouwanttohaveviolationnotificationssentbacktoyourservice,youcan

uncomment the reportUri setting and then handle that Express route in your

code.Refertothedocumentationforsomesamplecodeforthat.Helmetcannot

be your only mitigation for security threats. You need a thorough analysis of

your data flow diagram to come up with every threat and start building your

securityplan.

TheNodeSecurityProjectinitiative

ThereisanongoinginitiativetoauditthecodeofNPMmodulesandkeepa

databaseofknownvulnerabilities.Thisisbynomeanscomprehensive,butitis

goodthatthisisunderway.StartingwithNPMversion6,anyinstallyoudowill

automaticallygetverifiedtotellyouifthereareanyknownvulnerabilitieswith

theNPMmodulesyouareusing.Youcanalsorunacommandtohaveanaudit

run:

npmaudit

Makesuretoexecuteitinthefolderwhereyourpackage.jsonfileis.

11.5UnderstandingNodeInternals

YounowknowenoughtouseNodewithoutknowingmoreaboutitsinternal

workings.Youarethusfreetoskipthissectionifyoulike.Youmay,however,

wanttocomebacktothissectionlaterasitmighthelpclearupsomeadvanced

questionsthatmightarise.

Note:ThissectionwaswrittenfromwhatIlearnedbyreadingthroughthe

actual source code for Node.js found on GitHub. Go to

https://github.com/nodejs/nodeifyou are interestedin the internalworkings of

Node.jslikeIwas.LibuvcodeisincludedintheNode.jssourcecode.Youcan

findoutmoreaboutitsAPIifyougotohttp://libuv.org/.

ThelayersofNode.js

You can see from the following block diagram that there exists a main

Node.jscodebasewithdependenciesbelowthat.Thetoptwolayersofcodeare

whatmakeuptheNode.jsframework.Thebottomtwolayersaredependencies

thatNode.jsrelieson.

The very top layer is the JavaScript library that you will be using directly

from your code. Any time your code does something that is outside of the

standardJavaScriptcalls,youwillendupusingsomethingthatisfoundinthis

layer. For example, all of the following code is made possible by the library

layer:

varhttp=require('http');

varserver=http.createServer(function(request,response){

response.writeHead(200,{"Content-Type":"text/plain"});

response.end("HelloWorld\n");

});

server.listen(3000);

This top layer is what is documented on the Node.js official site

https://nodejs.org/en/docs/asitsAPI.ToreallygetstartedanduseNode,thatis

the only layer you are really required to know anything about. All core Node

modulesareexposedthroughthislibrarylayer.

HerearethefourlayersthatcomprisetheoperationofNodeasaframework

runtime.Itispossiblethatthisdiagram mightchangeasNode.jschanges over

time.

Figure46-Node.jsplatform

ItwouldbegreatiftheOSitselfcouldunderstandJavaScript,butitdoesnot

havethelibrariesexposedforislikeitdoesforC/C++.Thus,thereneedstobe

some translation from your code to code that the operating system can

understand.Thisiswhythebindingslayerisneeded.

The Node.js C bindings layer is made up of C++ code. This is what takes

yourcodethatisinJavaScriptandallowsittocalldowntocodelibrarieslike

LibuvthatarewritteninC.

NodeusesV8asadependency.Itdoessofortwomainpurposes.V8makes

itpossibleforyourJavaScriptcodetocallthroughtoC++coderunninginthe

Node.js process. Take the following example of some Node.js JavaScript

application code you might have. This code displays the size in bytes of your

package.jsonfile:

varfs=require('fs');

fs.stat("package.json",function(error,stats){

console.log(stats.size);

});

ThewayitworksisthatNodehasexposedthefsmoduleinthelibrarylayer

thatisbeingusedhere.NodeusessomecapabilitiesofV8toactuallytakethe

fs.stat()call and have that make a call to a C++ function in the bindings layer

calledStat().ThisC++functioninthebindinglayerthenmakesacalltoLibuv,

whichinturncallOSappropriatelow-levelcode.Eventually,itmakesitswayto

aUnixflavorOSlibrarycallofstat()oronaWindowssystem,thecallmadeis

NtQueryInformationFile()thatisexposedin ntdll.dll.Thecallbackthenmakes

itswaybacktoyourJavaScriptwheretheasynchronousoperationcompletes.

V8actsasaVirtualMachineinthesensethatitcanisolateandexecutesome

codeandbehostedmanytimesononemachineindependently.Itprovidesallof

theaspectsnecessaryforalanguageruntime.YoucanreadanintroductiontoV8

JavaScriptengineby going tohttps://developers.google.com/v8/intro#about-v8.

Hereissometextfromthegooglesite:

“V8 is Google's open source, high-performance JavaScript engine. It is

writteninC++andisusedinGoogleChrome,Google'sopensourcebrowser…

V8 compiles and executes JavaScript source code, handles memory allocation

forobjects,andgarbagecollectsobjectsitnolongerneeds…V8does,however,

provide all the data types, operators, objects and functions specified in the

ECMAstandard.V8enablesanyC++applicationtoexposeitsownobjectsand

functionstoJavaScriptcode.”

BesidesLibuvandV8,thereareafewotherdependenciesthatNodeusesfor

operationsasnotedinthediagram.Toactuallylookatwhatthedependenciesare

ofNode.js,youcangototheGitHubprojectandlookinthedepsfolder.Node.js

isveryportableasitsdependencieshavebeenportedtomanyplatforms.

Youhavenowseenhowallofthelayersfittogether.YouseethatJavaScript

codecanbeexecutedandthatfunctioncallscanmaketheirwaythroughNode

modules.SomeofthosemodulesinvokecodethroughthebindinglayerinC++

downintoLibuv.

Thenextthingtounderstandishowtheprocessingloopcomesintoplay.I

leftoutexplainingthisthusfar,butitisimportanttounderstand.Understanding

theprocessingloophelpsyoutobeawareofwheretheprocessingtakesplace

andhowimportantitisforyoutokeepyourasynccallbackcodeasperformant

aspossiblesoastokeepthemainthreadfree.

OperationofNode

Do not believe everything you read about Node.js. Some people are under

the misconception that Node.js only has a single thread and can only do one

thingatatime.Thisissimplynottrue.Iwilldispelthatmisconceptionandteach

youexactlyhowNode.jsexecutes.

Nowthatyouhaveseenthelayersofcodethatyourapplicationsitsontop

of, you are ready to see how these layers actually operate to orchestrate the

executionflowofaNodeapplication.ThemainoperationofNodeisillustrated

in the following diagram. Not every dependant component has been included,

justthemainonesconcerningprocessingflow:

Figure47-Node.jsconceptualarchitecture

It is true that Node.js has a single-threaded architecture for its main

JavaScript processing. What is mostly executing on that thread are your

JavaScriptfunctionsthatNodeexecutesinresponsetoevents.Theseareinthe

formofasynchronouscallbacks.Thisiswhatgivesyouconcurrentprocessingin

yourapp.MeaningthatmorethanoneoperationisexecutinginthegutsofNode

andontheoperatingsysteminthebackground,butonlyonereturncanandbe

processedinyourcodeatatime.

Internally, Node can actually use multiple threads to shuttle work off to.

Node.js utilizes a callback code pattern which frees up the developer from

havingtomanagethethreadingandpollingthemselves.Forexample,youmake

your call for retrieving data from a backend database, and as part of that call,

you provide a callback function. Your code immediately returns and then

Node.js takes care of knowing when the data is returned and schedules your

callbacktorunatalatertime.Yourcallbackisthusrunasynchronously.

Node.jsdoes not have any sleep, mutex lock, or any similarfunctions that

youmightseeinotherruntimes.OnthelowerlayersofNode,suchasinLibuv,

thethreadsofNodecanbeexecutinginparallel,takingadvantageofrunningon

amulti-coremachine.TheOSwillbeabletotaketheexecutingthreadsofNode

and spread those out across the cores that are available. Thus, things like file

systemoperationsandnetworkrequestsdohappeninparallel,eventhoughyour

processingoftheresultscanonlyhappenserially.

Your JavaScript callback code is executed asynchronously, meaning you

makethecallanditisnotblockingandcodeexecutioncancontinueelsewhere

andyoudon’tknowwhenthereturnwillbeprocessed.Itisconcurrentbecause

yourJavaScirptcodecancallmanydifferentasynchronousAPIsandhavethem

allinaction,orqueuedwithoutyouworryingaboutthem.Itisparallelbecause

theOSwilltakethemanyprocessingthreads,orlow-levelcallsthatNodemakes

andspreadthemoutacrosstheavailableCPUcores.

Ifyou gobackand considerthe code sample that used the fs module, you

saw that it eventually made it all the way down to the lowest level OS stat()

function. That call is obviously a blocking call! This is where Libuv does the

work for you to queue up that request to take it off your thread and return

immediately. Libuv will then run the stat() call on a thread in its own thread

pool.Whenthecallfinishes,Libuvknowsthecallbacktocallandsendsitback

uptobeexecutedinV8.

The thread pool threads of Libuv are being used over and over. There are

fourofthembydefault,butyoucanchangethattobemoreifyoufindaneed.

The NewsWatcher sample does not use the filesystem directly, so it does not

alter that. The Net module ends up being used by the NewsWatcher database

interactionsandthatdoesnotusetheLibuvthreadpool.

BootstrappingofNode

If you look back at figure 47 you can see the upper left part has what is

termed the bootstrapping of Node. This is something that Node sequences

throughtogetupandrunning.Hereisthegeneralsequenceandexplanationof

thediagram:

1. The Node process is run from a command line and has a main()

functionthatiscalledasitsprocessentrypointinitsC++code.

1. Themainfunctioncreates theruntimeenvironment and

thenloadsit.

2. Anobjectcalled“process”iscreatedthathasproperties

andfunctionsonit.Thisobjectisveryimportantandisused

throughoutthecode.

3. A JavaScript file is now run in the V8 VM that

bootstrapsthewholeprocess.

1. Some internal Node modules are parsed and

madeavailable.

2. The file (i.e. server.js) that node was started

withasanargumentisreadandruninV8.

2. Yourserver.jsfilerunsandatthispointcanusethefullcapabilities

ofNodetodothingslikeuseJavaScript,setuptimers,setupHTTP

listeners,makeHTTPrequests,setupmiddlewareetc.

3. After your server.js code is finished being processed, the Node

processentersitsperpetualprocessingloop.Atthispoint,theprocess

keepsrunningaspreviouslystatedaslongasthereisworktoprocess.

1. Libuvprocessesworkonitsqueuewithitsprocessloop

andthreads.

2. Callbacksmakeitbacktothe mainthreadtoberunby

theV8VM.V8wouldqueuecallsthatcomeinandprocess

them.

Forthoseof youthatare stillskepticalabout mystatingthat therearetwo

processingloops,hereistheproof.Hereisthecodefromnode.ccthatthemain()

functioneventuallyentersandkeepsexecuting.Thisisreallywhatcanbecalled

themainprocessingloopofNode:

boolmore;

do{

v8::platform::PumpMessageLoop(default_platform,isolate);

more=uv_run(env->event_loop(),UV_RUN_ONCE);

if(more==false){

v8::platform::PumpMessageLoop(default_platform,isolate);

EmitBeforeExit(env);

//Emit`beforeExit`iftheloopbecamealiveeitherafteremitting

//event,orafterrunningsomecallbacks.

more=uv_loop_alive(env->event_loop());

if(uv_run(env->event_loop(),UV_RUN_NOWAIT)!=0)

more=true;

}

}while(more==true);

NotehowtheLibuvprocessingloopisnotallowedtocontinuallyrun,butis

calledandisunderthecontrolofthemainloopofthenodeprocess.Themain

Node process loop calls libuv to let it run with the UV_RUN_NOWAIT flag.

Themainloop thusiscontinuallycallingtohave theLibuvloop runoverand

over.HereistheUnixportedversionoftheLibuvprocessingloop:

intuv_run(uv_loop_t*loop,uv_run_modemode){

inttimeout;

intr;

intran_pending;

r=uv__loop_alive(loop);

if(!r)

uv__update_time(loop);

while(r!=0&&loop->stop_flag==0){

uv__update_time(loop);

uv__run_timers(loop);

ran_pending=uv__run_pending(loop);

uv__run_idle(loop);

uv__run_prepare(loop);

timeout=0;

if((mode==UV_RUN_ONCE&&!ran_pending)||mode==UV_RUN_DEFAULT)

timeout=uv_backend_timeout(loop);

uv__io_poll(loop,timeout);

uv__run_check(loop);

uv__run_closing_handles(loop);

if(mode==UV_RUN_ONCE){

uv__update_time(loop);

uv__run_timers(loop);

}

r=uv__loop_alive(loop);

if(mode==UV_RUN_ONCE||mode==UV_RUN_NOWAIT)

break;

}

/*Theifstatementletsgcccompileittoaconditionalstore.Avoids

*dirtyingacacheline.

if(loop->stop_flag!=0)

loop->stop_flag=0;

returnr;

}

You can see that the while loop has code to cause a break statement to

happeninthecaseofNodecallingit.

Callback functions in your JavaScript usage of modules such as with

fs.reafFile() are kept by Node in a structure. When the low-level Libuv call

returns,thecallbackhappensontheC++mainthreadofNode.js.Ifyoulookat

thediagramagain,youcanseethereisatwo-wayarrowfromLibuvtotheV8

VM.

The call coming down from JavaScript through the C++ bindings uses the

V8 FunctionTemplate class to accomplish this. The callback going up from

Libuv makes it back to the C++ bindings layer code and uses the V8

Function::Call() function to have the V8 VM execute the actual JavaScript

callbackyouhadprovided.

ThiscallbackisnotexecutedontheNodeprocessthreadthatisorchestrating

all of this, but is executed in the V8 VM. Node is in no way managing any

queue,eventorprocessingloopfortheJavaScriptexecution.Thisisalldoneby

V8.

Theeffectofcomputeintensivecode

Themaineventthreadshouldonlybeusedtodofast,lessintenseprocessing

for inbound or outbound results. Take the following example that shows

processingthatisunacceptable:

varhttp=require('http');

varbcrypt=require('bcryptjs');

varcount=0;

varserver=http.createServer(function(request,response){

for(i=0;i<=10;i++){

bcrypt.hashSync("hjkl5678jhg",10);

}

response.writeHead(200,{"Content-Type":"text/plain"});

response.end("HelloWorld\n"+count++);

});

server.listen(3000);

Everywebrequest thatcomes inwill causethis compute-intensivecodeto

run.Thiswouldbasicallydevourallthecomputetimeasrequestskeepcoming

in.Youreventloopwouldnothaveanytimetoprocessanyincomingrequests.

The point here is that you should never do CPU intensive calculations in

yourmainNodethreadcallbackcode.Ifyoudo,itwillpreventtheentiresystem

fromworkingcorrectly.

Youcan solve this problem in a simple way with Node. There are built-in

capabilitiestosendexpensiveprocessingtoseparateforkedprocessesthatdonot

affectyourmainNode.jsprocess.Processingcanbeforkedtochildprocessesor

eventoaseparateexecutablerunningonyourcomputerthatyouhavewrittenin

anotherlanguage.YoucouldqueueupworkinacentralqueueandhaveNode

workerprocessespullfromthat.

Youcanalsocreateanadd-oninC++forNode.js.Youmightconsidersome

remoteservicecallsifyouhavecapabilitiesaccessiblethatway.Therearemany

creativethingsyoucandotosolvethisproblem.Asanotherpossibility,youcan

useAWSLambdatoaccomplishtheoffloadingofspotprocessingorotherwise

computeintensivecode.

In NewsWatcher we will be forking off code that does the fetching of the

global list of news stories. I only do it this way for now to simplify the

provisioning and deployment. The code that is running in the forked process,

couldbetakenoutandrunusingAWSLambda.

11.6HowtoScaleNode

CouldtheNode.jsprocessevergetoverloaded?Forexample,let’ssay1,000

HTTPGetrequestscametoyourNode.jsserverallatonce.Maybethis,inturn,

requires1,000filesbeingreadfromdisk.TheansweristhatNodeissetuptobe

abletomanageasizableworkload,soitmaydojustfine.Youwouldneedtodo

someprofilingforyourparticularworkload.

With few exceptions, Node.js makes your requests in a way that is non-

blocking.Forfilesysteminteractions,therequestsareshuttledofftothreadsin

theLibuvthreadpooltoruninparallel.Eachofthosedotheirworkandreturn

backtotheeventloopforcallbackexecutionscheduling.Asthreadpoolthreads

getfreeduptheyaregivenmorework.

Eventually,allofthe1,000filereadrequestswouldcomplete.Thishappens

intheleastamountoftimepossibleandalsowiththeleastamountofresources.

Allofthiscanbehandledquiteefficientlybyonesinglenodeprocess.

You can visualize this queue handoff as shown in the following diagram.

Note that the event loop of the Node process simply sequences through

everything in its queue if more come in than can be handled. You could

configurethroughcodetohavemorethanfourthreadsifyoulikeandyoucould

experimenttoseewhatperformanceimprovementsyoumighthave.

Figure48-Threadpoolhandoffforfilesystemrequests

Resultsarealwaysinterleavedandprocessedonebyoneonyoursinglemain

thread. As always, the performance is affected by how your JavaScript code

processestheresults.Thatiswherethebottlenecksalmostalwayshappen.

RememberthatnetworkI/Oishandleddifferentlythanfilesystemcallsand

doesnotusetheLibuvthreadpoolqueueaswasdiscussedforfilesystemusage.

AnyHTTP calls in your Node codeare handled by low level OSmechanisms

thatcanscale.

BeawarethattheNewsWatchersampleapplicationwillbeconnectingupto

aMongoDBdatabaseinthedatalayer.Thismoduleiswrittentouselow-level

TCPcallstotheMongoDBservice.Themoduleyougothroughtomakethose

calls uses the Net module of Node which eventually calls the library code of

Libuv to be making platform network calls. The MongoDB module pools

databaseconnections.Thismeansyouhaveacertainamountofconnectionsto

use for all the calls in your Node.js code and calls are queued and rotated

throughasconnectionsarefreedupfrompreviouscalls.Youcanconfigurethe

numberofconnectionsinthepool.

There is obviously a limit to how many operations you can achieve per

second before performance starts to degrade. Perhaps you have network I/O

requests going off to some slow-returning MongoDB call. The requests could

growoutofcontroliftheyarenothandledfastenoughandtheDBconnection

pool requests start to queue up. Of course, you also need to provide for

scalabilityinyourdatalayer,orthatwillbecomethebottleneck.

Note:At somepoint, Nodehas torelyon theOS andhardware(disks and

networkcards).Remember,thatyouhaveothersystemcontentionstobeaware

ofsuchasdiskcontrollercontentionsfromotherprocessesrunningonagiven

machine.

MultipleNodeprocessespermachine

Eventually, you can move to other architectural variations to handle your

load.Toscaletoagreatercapacity,youcanstartupmultipleNode.jsprocesses

on the same machine and distribute the load across those. If you have lots of

coresona machine,youcanmake useofall ofthemwith aNodeprocess for

eachcore.

TodothisscalingofNodeacrossthecoresofasinglemachineyoucanuse

the cluster module of Node.js itself to do the load balancing. There is also a

processmanagernamedPM2thatyoucandownloadfromNPMtodotheload

balancing. PM2 has other capabilities such as monitoring, restarting of Node

processes,andrunningrollingdeployments.

If you deploy your Node.js application through an AWS service, such as

ElasticBeanstalk,youcanutilizethepowerofPaaStoscaleeverythingforyou

byscalingupthenumberofcoresonanEC2VM(knownasverticalscaling)of

scaleupthenumberofEC2instances(knownashorizontalscaling).Eventually,

youwillnotbeabletohandleallrequestsandprocessingonasinglemachine.

Thsiswhenyoumuststartemployinghorizontalscaling.

SOAArchitecture

TheSOA(ServiceOrientedArchitecture)canbeunderstoodbycontrastingit

withwhat can beconsidered a monolithicarchitecture. Perhaps you haveseen

Java-basedorASP.Netwebservers.ThisiscodethatrenderstheHTMLonthe

server and sends it back to the client. It could have a lot of code that is

intertwinedtospitoutthisHTML.

Thistypeofmonolithicapplicationhasmanyshortcomings.Forexample,if

therewasachangetoonesinglepartofthecode,thewholeapplicationneedsto

bedeployedagain.Monolithiccodeistypicallytightlycoupledandveryfragile.

Themonolithendsupbeinghardtotest,enhanceandmaintain.

MovingtoaSOA,meansprovidingalayerthattheUIcouldaccesstogetat

functionalityasasetofservices.ThiscouldbeUIthatisserver-sideaswellas

from a client-side SPA. A great practice that came along was to create these

services as HTTP/REST web services that used JSON as the data transfer

format.

SOAwebservicesaresmallunitsofcodethatdoonethingandaretestable.

A Node.js application can use Express and have a nice RESTful approach

throughitsexposedpathsintheURLthatgoestotheroutehandlers.Thisends

upbeingaSOAwithallofthesebenefits.Yougetthebenefitsofsplittingout

thecapabilitiesintoseparatecodebases tobeseparatelydeveloped,testedand

maintained.Multiplepeoplecanworkoneachindividualserviceindependently

andeachroutehandlerisindependent.Eachservicehastheirownlogicandcan

go to their own database. They can actually go to the same database, but just

haveseparatedocumenttypesinsideofthatshareddatabase.

The code patterns for this type of SOA is what is implemented in the

NewsWatchersampleapplicationandcanachievesuperiorqualityandscaling.

You could deploy the same Node.js application across a horizontally scalable

cluster of machines. You could argue that you still have at least one of the

characteristicsof a monolith. Asingle codebase that has multiple services still

needs a complete deployment for each change that happens in its individual

services. The nice thing that makes it different is that each service really is

independentandisnotaffectedbyachangeinanotherservice,soitshouldnot

reallymatterifanewdeploymentofthecollectionofrouteshappens.

Scalinghorizontally

Let’s now discuss horizontal scaling. Perhaps you have four cores per

machineandhaveyourmainNodeprocessononecore.Youcancountonthe

OSmakinguseofallcoresforyouforoperationssuchasfilesystemoperations

thatusetheLibuvthreadpool.

Youcould employ the technique of forking processes from the main Node

process to offload some heavier computations and to do periodic batch

processing. If you are doing this, then you don’t have the spare cores to use

throughclusterorPM2usage.Thebetterapproachistospreadouthorizontally

acrossmachines.Youmustdothisanywayforreallyhighscaleneeds.

Scaling across machines is easy to configure in AWS. If you use Elastic

Beanstalk,itusesNginxasareverseproxyandloadbalancerforyourNode.js

application. Elastic Beanstalk lets you configure how many machines to load

balance between. You can also set up Elastic Beanstalk to use an auto-scaling

groupinsteadofusingindividualEC2instances.

Figure49-ElasticBeanstalkloadbalancing

EachEC2machinewouldalsobeinadifferentavailabilityzonetogiveyou

redundantfailoverifneeded.Thisisinadditiontotheloadbalancingbenefitthat

gives you more capacity. Figure 49 shows what that looks like with two EC2

VMinstances.

One nice thing with AWS Elastic Beanstalk, is that when machine OS

upgradesorpatchesneedtohappen,itisdoneonemachineatatime.Machines

are taken out of rotation, updated and then put back into rotation. When you

needtoupdateyourNode.js application,yourdeploymentalsohappens inthis

exactsamewaywitha“rollingupdate”.Thiswayyouachievewhatiscalleda

“zerodowntime”deployment.

Note:Beawarethatwithanyofthesescalingmechanisms,youmustensure

thatyouarerunninginacompletelystatelesswayandavoidanyaffinitysettings

to truly be able to distribute calls across all Node processes. To manage a

connectionrequiringstate,youcaninserttheuseofsomethinglikeRedistofetch

stateifneeded,orkeepstateonlyontheclientorinatokenthatispassedback

andforth.

Staticresourceserving

IntheNewsWatchersampleapplication,theNode.jsapplicationactstoserve

uptheHTTPrequestsfortheAPIroutes.ThesearetheRESTfulwebservices

thattheReactUIconsumes.TheNode.jsapplicationservesuptheactualReact

website.Thesearewhatarecalledstaticresources,becausetheydon’tchange.

All files, such as images, css, html etc. required for the React application are

containedinacompletelydifferentdirectorythatNodeknowsaboutandserves

up.Forexample,hereistheloggingthathappenswhentheReactapplicationis

requested.YoucanseetheJavaScriptbundledfileforReactthatisbeingsent.

The 304 code you see means that a resource was not send since the version

requestedwasstillthelatest.

Figure50-Staticresourcereturns

TherearemoreefficientwaystoserveuptheReactapplicationandfreeup

theNode.js application fromthat work. Forexample, the Nginx reverseproxy

load balancer can actually know about those requests and serve them up.

Another way would be to use something like Amazon S3 storage and set up

AWSRoute53togothroughAWSCloudFrontthatpullstheReactapplication

from the S3 storage. This is what is referred to as a CDN (Content Delivery

Network).

In the case of Nginx or CloudFront, they both support caching. Since the

React application resources are not updated often, requests would be fetched

fromthecacheandbeextremelyfast.Express.staticdoesnotdoanycaching,but

you can use client-side caching with ETag or Max-Age. By default, the static

middlewarehasETagsenabled.

ADockerContainerforyourNode.jsprocess

It is possible to use Docker as a means of deploying and running your

Node.js application. Elastic Beanstalk can even use a self-contained Docker

container.Yougetthesameprovisioning,loadbalancingandscalingyougetby

justdeployingastraightNode.jsapplication.

Youmightwanttotakethisapproachifyoufindthatyouhaveotherthings

to install alongside the Node.js application that can all be contained in the

Dockercontainer. If allyou have isyour Node.jscode to deployusing Elastic

Beanstalk,youmightnothaveanybenefitinusingDocker.

Itcanbeconvenientfordeveloperstobeworkinginthiswaysothatthereis

some structure and predictability as to what is needed for a deployment. This

wouldmeanthatifyouhaveaDockercontainerallrunningandtestedonyour

developmentmachine,itismorelikelytoalsofunctioninproduction,oratleast

notbemissinganythingthatneedstobeinstalledalongsideit.

Note:You can keep your Node.js application Docker images in AWS ECR

(EC2ContainerRegistry).

MicroservicesArchitecture

We discussed that implementing SOA was a great idea and that we have

achievedthatwithourRESTfulapproach.Aswasmentioned,theNewsWatcher

sample application keeps all of the RESTful routes in the same application.

ThinkaboutwhatwouldhappenifeachindividualHTTP/RESTroute(i.e.auth,

user,etc.)weresplitintotheirownNode.jsapplicationanddeployedseparately.

This gets us into the topic of realizing an architecture that is called a

Microservices architecture. The following figure shows the before and after

visualization of how our NewsWatcher application could be made into a

Microservicesarchitecture.

Figure51-Microservicesclusterdeploymenthosting

Each different background fill pattern in the diagram would represent a

differenthostedserviceendpoint.Forexample,Billing,UserProfile,Inventory

and such. Each can be independently scaled and the load is balanced across

availablemachines inthe cluster. Amazonhas ECS forthis andalso EKS and

Fargate.

Microservices are independently deployed and run and each has their own

API,middlebusinesslogicandbackenddatalayerasnecessary.Thedatacould

bedocumentsinasharedMongoDBcollection.Microservicescanbethoughtof

asaspecificwaytoimplementSOA.

ThetermSOAcertainlysurfacedmanyyearsagoanditsimplementationsat

thattimewereveryformalandburdenedbyspecificationssuchasSOAPWS-*,

XML/WSDLand were sometimes implemented ontopof an ESB(Enterprise

ServiceBus).Thatevolvedovertime,buttheoriginalconceptsarestillvalidin

today’smodernHTTPRESTandJSON.

Before rushing into a Microservices architecture, you should be sure you

reallyneedoneinthefirstplace.Implementingafullbuildanddeploysystem

formicroservicesismorecomplicated.Alwayskeepthingsassimpleaspossible

foryourneeds.Let’s lookatafewreasons peoplegiveforwhyMicroservices

aresogreatandunderstandunderwhatcircumstancesyoumightwanttoattempt

this.

One benefit is that services are split out into discreet pieces that are much

smaller.Theargumentisthatsmallerpiecesareeasiertobuildandtest.Youmay

haveheardofcompaniesthatmanagethousandsofmicroserviceatonce.Small

Microservicesprovideallofthebackendservicingforanynumberofconsumers

on the front end. Amazon is one such company that utilizes Microservices.

Chances are that you are a much smaller company than Amazon and your

applicationsaremuchsimpler.

Remembertodowhatisrightforyourneeds.Buildyourbackendservicesat

a granularity that makes development and management easiest for your

operation.ItisunfortunatethatthetermMicroservicesstartswith“micro”.This

makesone thinkthatthey mustbe really small.Indeed, some peopleadvocate

microservices that are 100 lines or less. If you really want to do that, then I

would advise you go to a serverless architecture and not really do Node.js

application,buttolookatthingslikeAWSLambdaFunctions.

Ifyouwerewritingabackendbillingservice,youcanimaginethatitcould

be a large amount of code. You can initially try to split it out into smaller

consumableunitsthatcouldeachbeusedindependently.Iftheycannotbeused

independently,thenitshouldbeonecohesiveservice.Atonepoint,astatement

was made in a presentation by Amazon on Microservices that the main

amazon.com page makes 100 to 150 backend calls to Microservices at initial

loadtime.Theseareverysmallconsumableservices.

Anotherbenefit that is always broughtup is that you can scalebetter with

microservices.Thatisofcoursedebatable.IfyouweretotaketheNewsWatcher

applicationasitisanddeploythatonafewdozenmachines,youcouldscaleto

tensofmillionsofcallsaday.Whatwouldhappenifyousplitouttheroutesinto

theirowndeployableNode.jsapplicationservicesanddeployedthoseacrossthe

same machines? You might actually get poorer performance because each

machine would be running a set of Node.js processes instead of just one.

Obviously, the argument is that not all services would be deployed across all

machines in the same amounts. Services like billing, might not get called as

oftenandcouldbedeployedonafewmachines,whileaproductlookupservice

mightbecalledalotmoreoftenandneedtobedeployedontolotsofmachines.

Thereasonyouwouldwanttosplitoutandruntheseparateserviceswould

reducedowntothefollowingobservations:

1. You have completely autonomous teams (even geographically

separated)thatdon’tknowabouttheotherservicesteamsandeachis

creatinganddeployingservicesindependently.

2. Eachservicegetsdeployedonthenumberofmachinesrequiredto

handletheloaditneedstosupport.

Youwould monitor the transactions and loadingneeds of eachserviceand

deploy them out to your cluster as needed. The point of a sophisticated PaaS

solutionforaMicroservicesarchitectureisthatitwouldhandlethisdistribution

andshufflingofservicestobalancethemacrossmachines.

Microservicesframeworks

If you have split up your Node.js application into several smaller

applications,eachneedingtheirown“npmstart”commandtorun,thenyouwill

wanttoconsiderusingDockerforeachofthesediscreetNode.jsapplications.To

do that, you will want to use something like Amazon ECS, EKS, or Fargate.

ThereisalsoOpenShiftOnline,Kubernetes,DockerSwarm,CloudFoundry,or

NGINXUnit(ApplicationServerandServiceMesh).Theseframeworksmanage

thedeploymentofyourDockercontainerstoaclusterofmachines.

Really sophisticated Microservices architectures may even employ some

typeofservicediscoveryregistry.ThereareframeworkssuchasScenicathatare

availabletodothis.Youneedtojustifythatyoureallyneedthiscomplexityand

overhead.

ThereisaNode.jspackagenamedHydrathatyoucandownloadfromNPM

that might be of benefit to you is you are creating a more complex

implementation. Hydra gives you things like service discovery, distributed

messaging,messageloadbalancing,logging,presence,andhealthmonitoring.

Queuesandworkerprocesses

A common approach in backend services is to take requests that require

longer running processing and accept that request immediately and return

success,butactuallyjustqueueitup.Imagineyouhadaphotostorageservice

that has an HTTP Post endpoint API that accepts images. You might want to

createthumbnailimagesanddosomeimageprocessingtodofacialrecognition

andaddmetadatatoidentifypeople.

Tokeepyourservicescalable,youcouldhaveyourwebserviceacceptthe

image,copyittoS3andthenplaceanentryintoMongoDBorAWSSQStobe

lookedat later. Then youcould have separateEC2 machinesor AWSLambda

functions that scale and independently read the work requests and do the

processing. This is how to manage the work and keep the web service very

responsive.

Chapter12:NewsWatcherApp

Development

Itis now timeto beginconstructinga Node.jsService layer thatintegrates

withtheMongoDBdatalayerfrompartoneofthisbook.Thischaptertakesthe

concepts you have already learned regarding Node and applies them in a real

project.WhatyouwillbecreatingisaRESTfulwebAPIthatyourpresentation

layerwillbeabletointegratewith.Youwilllearnhowtoimplementeverything

needed for a fully functional cloud-hosted web service. You will utilize best

practicesfortestingandDevOpsinthechaptersthatfollow.

Note:Don’tforgetthatyoucanaccessallofthecodefortheNewsWatcher

sampleprojectathttps://github.com/eljamaki01/NewsWatcher2RWeb.

12.1InstalltheNecessaryTools

OneoftheamazingthingsaboutNodeishowsimpleandquickitistosetup

aserver.ItcanbehostedinAWSasaPaaSofferingusingElasticBeanstalk,or

otherhostinginfrastructure,orevenbedeployedandmanagedmanually.Node.js

runsonmanydifferentoperatingsystems.

Togetstarted,installthefollowing:

✓AcodeeditorsuchasVisualStudioCode,Sublimetext,Vim,etc.

✓ Node.jsfromhttps://nodejs.org/.Getastableversion.Thisinstallsthe

nodeexecutableforyouandalsoinstallstheNPMexecutable.

Note: Visual Studio Code is not the same as Visual Studio. VS Code is a

completelynewtoolthatoffersaricheditingenvironmentaswellasintegrated

featuresforsourcecodecontrolanddebugging.VSCodehasthecapabilityto

launchtasksthroughthemeansoftoolslikeGulpwithnoneedtojumpouttoa

commandline.Thesetoolscanbeusedtoautomatebuildandteststepsthatyou

need to run frequently. With Visual Studio Code, you will be able to create a

project and run it locally on your machine and have access to IntelliSense,

debugging,andwebapppublishingthroughGit/GitHub.

12.2CreateanExpressApplication

Start by creating a folder for your application. I named mine

“NewsWatcher2RWeb”. You can now create the minimum amount of code

requiredforaNode.jsapplication.Tohelpmaintainyoursanity,youshouldstart

withthesmallestamountofcodepossibleandpushitallthewaytodeployment.

This will eliminate unneeded investigation time of issues unrelated to just

gettingthebasicserviceupandworking.

Launch Visual Studio Code and click File->Open Folder, then select the

“NewsWatcher2RWeb”folderyoujustcreated.WithVSCodeopen,youseethe

EXPLORE view open and in there find two subfolders. You can also use a

commandpromptandtype“code.”toopenupVSCodeintheprojectdirectory.

Figure52-VSCodeUI

TheNewsWatcher2RWebfoldershowsyouallfilesandsubfolders.

Youcan now get started and create a simple Node.js application, and then

deployittoanAWSElasticBeanstalkEC2instance.Oncethatisverifiedand

runningok,youcanaddmorecodetofilloutthefullfunctionalityoftheweb

service.

On the NEWSWATCHER subfolder, click on the icon to create new files.

Startbycreatingthesethreefiles:

.gitignore

package.json

server.js

The.gitignorefilewillnotbeusedrightnow,butwouldcomeintousewhen

you make use of Git and GitHub. You list files and directories you want to

excludefromGitsourcecodecontrol.

Atthis point,there is nonode_modulesfolder yet.It will automaticallybe

createdwhenyouinstallthenodemoduleswiththe“npminstall”command.

Addthefollowingcodetotheserver.jsfile:

varexpress=require('express');

varapp=express();

app.get('/',function(req,res){

console.log('Sendmessageongetrequest');

res.send('Hellofull-stackdevelopment!');

});

app.set('port',process.env.PORT||3000);

varserver=app.listen(app.get('port'),function(){console.log('Expressserverlisteningonport:'+

server.address().port);

});

Addthefollowinglinestothepackage.jsonfile:

{

"name":"NewsWatcher",

"version":"0.0.0",

"description":"NewsWatcher",

"main":"server.js",

"author":{

"name":"yourname",

"email":""

"scripts":{

"start":"nodeserver.js"

"dependencies":{

"express":"^4.13.4"

}

Saveallthefiles,thenopenacommandpromptwindowandnavigatetoyour

project’sdirectory.Atthecommandprompt,typenpminstall.Thiswilllookat

your package.json file, and install the standard Node modules along with the

Express module that is listed as a dependency. A new directory will be added

withthenamenode_modules.

You can now try running your Node project locally. Once it is proven to

function,youwillworkongettingitdeployedtoAWS.Atthecommandprompt,

type"npmstart".Youcanalsotype"nodeserver.js"torunit.Iftheprojectruns

successfully,youwillseethefollowingconsoleoutput:

Figure53-localconsoleoutput

Open a web browser and navigate to http://localhost:3000/. In the browser

window,youwillseeyourmessage:

Figure54-Projectmessageinabrowserwindow

12.3DeployingtoAWSElasticBeanstalk

It is now time to create your Elastic Beanstalk app through the AWS

Management Console. To do so, you must already have an AWS account. Be

awarethatyoumayincursomecostatthispointifyouarenotrunningwitha

freeaccountwithAWS!

Tocreatetheapp:

1. Open a web browser and navigate to

https://console.aws.amazon.com/console/.

2. In the upper right corner of the web page, for Region, select a

region such as US East (N. Virginia) as the region you want your

servicestoberunningin.

Figure55-Regionselection

3. Fromtheselectionofservices,clickElasticBeanstalk:

Figure56-SelectElasticBeanstalk

4. ClickCreate New Application. Enter a name and description. I

gaveitthename“newswatcher”.

Figure57-ElasticBeanstalkcreatenewapplication

5. Click to create a new environment by clicking Createone now.

SelectaWebserverenvironment.

6. SelectNode.jsastheplatformandclickCreateenvironment.

Oncethesiteisready,theelasticbeanstalkportallooksasfollows:

Figure58-ElasticBeanstalkportal

ClicktheURL:linkatthetopofthepagetoseeyoursiteworking.

Figure59-Launchthesite

Now you can deploy your own simple Express application.On a windows

machine,thisisasfollows:

1. UsingtheWindowsFileExplorer,navigatetoyourprojectfolder.

2. Selectthepackage.jsonandserver.jsfilestogether,thenright-click

andselectSendto->Compressed(zipped)folder.Givethezipfilea

nameandsaveit.

3. In the Elastic Beanstalk dashboard for the newswatcher

application,clickUploadandDeploybuttonandselectyourzipfile.

4. Wait for the confirmation that the deployment is ready and click

theURLagain.

YourNode.jsapplicationisnowworkingforeveryonetosee.

Youdonotneedtozipandsendthenode_modulesfolder.Thedeploymentto

ElasticBeanstalkwillrun'npminstall'foryouontheEC2instances.Onething

you should do is to set an environment variable through the Elastic Beanstalk

management console so that the node install actually becomes “npm install –

production”. Set the environment variable in the Configuration -> Software

Configuration->EnvironmentProperties.as:

Propertyname:NPM_CONFIG_PRODUCTIONPropertyvalue:true

This will make the install go much faster as it will not deploy any npm

modulesthatareneededintestordevelopmentenvironments.

Figure60-Uploadanddeploythezipfile

Youjustperformedamanualdeployment.Whileafewmanualdeployments

mightbe tolerable, you eventually want full continuous-integrationscripts that

runtestsanddeploymentsforyou.

12.4BasicProjectStructure

YoucannowaddtherestofthecodefortheNewsWatcherapplication.You

canstartby addinginthe restofthe Node.jsdependenciesthat youwillneed.

Edityourpackage.jsonfiletobeasfollows,thensaveit.Itshouldlooksimilarto

thefollowing.YoucanrefertotheGitHubprojectforthecompletefile.

{

"name":"newswatcher",

"version":"0.0.1",

"main":"server.js",

"scripts":{

"start":"nodeserver.js",

"dependencies":{

"async":"^2.6.0",

"bcryptjs":"^2.4.3",

"body-parser":"^1.18.2",

"dotenv":"^4.0.0",

"express":"^4.16.3",

"express-rate-limit":"^2.11.0",

"helmet":"^3.12.0",

"helmet-csp":"^2.7.0",

"joi":"^13.3.0",

"jwt-simple":"^0.5.1",

"mongodb":"^3.0.8",

"morgan":"^1.9.0",

"response-time":"^2.3.2"

"devDependencies":{

"eslint":"^4.19.1",

"eslint-plugin-react":"^7.8.1",

"mocha":"^4.1.0",

"selenium-webdriver":"^3.6.0",

"supertest":"^3.0.0"

}

Nowopenacommandpromptwindowandtypenpminstallatthecommand

prompt.Youwilladdcodetotherestofthefileslaterinthischapter.Youcan,of

course, go to the GitHub project and get all of the code for the NewsWatcher

sampleapplication.Youarenowreadytoputyourservicelayerwebapplication

RESTfulAPItogether.

Note:Ifyoueverwanttocompletelyupdatetothelatestversionofone or

moreinstalledpackagesinyourpackage.json,changeeveryversionnumberto

“*”,andthenrun–“npmupdate-–save”and“npmupdate–save-dev”.

12.5WhereitAllStarts(server.js)

ThestartingpointofyourNodeapplicationcanbeinafilenamedwhatever

youlike,Ipreferserver.js.ThisfileinstructsNodewhattodotobeinitialized

andwhattosubsequentlyexecute,onceupandrunning.Inreality,thisfilecould

benamedanything.ThisfiledoesthingslikeestablishaMongoDBconnection,

set up error handling, and establish the Express listener for the HTTP request

handling.

Atthetopoftheserver.jsfileiswhereyouwillplacetherequirestatements

thatloadtheneededmodules.Ifyourecall,modulereferencesinsideafileare

internaltothatfile.

Someofthemodulesyouwillspecifyasbeingrequiredareactuallyusedas

middlewarefor Express. This means you set them up with a require statement

andthendon’tactuallyusethemdirectly.Theyacttointerceptcallsthroughan

app.use()setting.Ifneeded,youcanreferbacktothechapteronmiddlewareto

reviewthisconcept.

Hereisthefirstsectionofcodeintheserver.jsfile.Ihaveaddedcomments

tobrieflydescribethepurposeofeachrequirestatement.

varexpress=require('express');//Routehandlersandtemplatesusage

varpath=require('path');//Populatingthepathpropertyoftherequest

varlogger=require('morgan');//HTTPrequestlogging

varbodyParser=require('body-parser');//AccesstotheHTTPrequestbody

varcp=require('child_process');//ForkingaseparateNode.jsprocesses

varresponseTime=require('response-time');//Performancelogging

varassert=require('assert');//asserttestingofvalues

varhelmet=require('helmet');//Securitymeasures

varRateLimit=require('express-rate-limit');//IPbasedratelimiter

varcsp=require('helmet-csp');

The first require statement provides the object that will be needed for

leveragingExpress.Thepathmoduleisbasicallyusedtoprovideahelperobject

that will be used to manipulate strings for specifying the file paths in your

project.Therestoftherequirestatementsareforsettingupmodulesthatactas

middleware.

Thesenextlinestoaddwillsetupenvironmentsettingsforthecodethatcan

havethingslikepasswordsandothersecretsandconfigurationsthatareneeded.

Whenrunlocally,thisisreadoutofafilenamed.env.Whenruninproduction,

the file is not used, but the values are set in the AWS Elastic Beanstalk

environmentsettingsandthecodeseesthoseinstead.

if(process.env.NODE_ENV!=='production'){//readinginofsecrets

require('dotenv').config();

}

Thesenextlineswillpullincodefrommodulesthatyouwillwriteyourself

thatprovidetheroutehandlers:

varusers=require('./routes/users');

varsession=require('./routes/session');

varsharedNews=require('./routes/sharedNews');

varhomeNews=require('./routes/homeNews');

Thecodecannowstarttoimplementsomeofitscapabilities.Younowmake

theExpressapplicationobjectavailableandsetasettingonitthatwillbeneeded

forwhenitisruninAWS.SincetheappisbehindanNginxloadbalancerwith

ElasticBeanstalk,youdon’twanttheloadbalancerIPaddresssentintheheader

requests, but want the IP address of the actual machine that it was acting on

behalfof.Thisiswhatthetrustproxysettingdoes.

varapp=express();

app.enable('trustproxy');

Middleware

Next,youcanseehowtheExpressmiddlewareishookedupforsomeofthe

modulesyouareincorporating.Herearethoselines:

//Applylimitstoallrequests

varlimiter=newRateLimit({

windowMs:15*60*1000,//15minutes

max:100,//limiteachIPto100requestsperwindowMs

delayMs:0//disabledelaying-fullspeeduntilthemaxlimit

});

app.use(limiter);

app.use(helmet());//Takethedefaultstostartwith

app.use(csp({

//Specifydirectivesforcontentsources

directives:{

defaultSrc:["'self'"],

scriptSrc:["'self'","'unsafe-inline'",'ajax.googleapis.com',

'maxcdn.bootstrapcdn.com'],

styleSrc:["'self'","'unsafe-inline'",'maxcdn.bootstrapcdn.com'],

fontSrc:["'self'",'maxcdn.bootstrapcdn.com'],

imgSrc:['*']

}

}));

//AddsanX-Response-Timeheadertoresponsestomeasureresponsetimes

app.use(responseTime());

//logsallHTTPrequests.The"dev"optiongivesitaspecificstyling

app.use(logger('dev'));

//Setsuptheresponseobjectinroutestocontainabodypropertywithan

//objectofwhatisparsedfromaJSONbodyrequestpayload

//Thereisnoneedforallowingahugebody,itmightbesomeattack,

//sousethelimitoption

app.use(bodyParser.json({limit:'100kb'}));

//MainHTMLpagetobereturnedisinthebuilddirectory

app.get('/',function(req,res){

res.sendFile(path.join(__dirname,'build','index.html'));

});

//ServingupofstaticcontentsuchasHTMLfortheReact

//SPA,images,CSSfiles,andJavaScriptfiles

app.use(express.static(path.join(__dirname,'build')));

Thiscodetakesthemodulesbroughtinthroughtherequire()statementsand

insertsthemasmiddlewarebycallingapp.use().

ThefirstpieceofmiddlewareiswhatgivesprotectionagainstDoSattacks.

You can look up the module in GitHub to actually see how it works as

middleware.

ThenextpieceofmiddlewareisHelmet.Icovereditsuseearlier.Helmetisa

securitymitigationmodulethattweakstheHTTPheaders.

Therearefiveotherusesofmiddlewarethataredocumentedinthecodeto

tellyouwhattheydo.Eachisveryusefulandyouwillbenefitfromthemall.

You can see the use of the path module to provide functionality to

manipulate path strings with the join function. The __dirname variable is

providedbyNodesothatyoucanuseittogetthenameofthedirectorythatthe

currentlyexecutingcoderesidesin.Inthisusage,itwouldreturnthedirectoryof

theserver.jsfile.Itwillbethelocalpathifyouarerunningitlocally,orwhatever

it is on the AWS production machine if it is running in the deployed

environment.

ThecodingoftheReactWebSPAReactiscoveredinthethirdpartofthis

book. You can see the code that serves up this static content that is using the

Expressstaticmodule.

ForkingaProcess

The next code in the server.js file is used to fork off a separate Node.js

process and give it a file to execute. This is used to shuttle off any code

processingthatismoreintensiveandthatyoudon’twantrunonyourmainNode

processthread.

varnode2=cp.fork('./worker/app_FORK.js');

Earlier,Iexplainedthatyouneedtobecarefulwithcodeyouwritethatwills

executeonthemainNode.jsV8VM.WithNewsWatcheryouneedtooffloada

fewthingstoaseparateNodeprocess.Theseinvolvecodeforcollectingnews

storiesfrom internet sources intoNewsWatcher’s masterlist in MongoDB and

doingthefilteredmatchingofstoriesforusers.Iwillshowyouthecodeinthe

forkedprocesslater.Rememberalso, thatIsaiditwouldprobablybebetterto

useAWSLambdaforthistypeofcodeprocessing.

You can see that you use the child_process module to start up the second

process.SinceNodeisportedtomanyplatforms,itwillcallwhateverlow-level

codeisneededtoaccomplishthisfortheOSitisrunningon.Nodemakesuseof

some libraries that do this and these have been ported already. You can pass

messagesbackandforthbetweenprocessesifyoulike.Youwillbedoingthis

laterincode.

Iftheforkedprocessisexperiencingruntimeerrors,itcouldshutitselfdown

andthenthemainprocesscouldbesignaledtostartitupagain.Youaddcodein

server.jstorestarttheforkedprocessasfollows:

node2.on('exit',function(code){

node2=undefined;

node2=cp.fork('./worker/app_FORK.js');

});

TheMongoDBDataLayerConnection

Mostofthecodeintheservicelayerdealswithinteractionswiththebackend

data storage layer. You initialize your MongoDB connection by utilizing the

mongodbmodulethatisanNPMdownload.Youusetheconnect()functionand

then set up the usage of the newswatcher collection. The connect() function

takestheMongoDBconnectionURL.

vardb={};

varMongoClient=require('mongodb').MongoClient;

//UseconnectmethodtoconnecttotheServer

MongoClient.connect(process.env.MONGODB_CONNECT_URL,function(err,client){

assert.equal(null,err);

db.client=client;

db.collection=client.db('newswatcherdb').collection('newswatcher');

});

Yousavetheconnectionasapropertyonanobjectyousetupnameddbto

beusedlaterwithyourExpressroutesthroughmiddlewareinjection.Watchfor

thatcodecomingupsoon.

Thelastthingtonoteabouttheabovecodeisthatyouhaveaconfiguration

file(.env)forkeepingsettingsinthatyouwanttohaveinacentralplace.Some

ofthevaluesinthatfileareonesyouwanttokeepsecret.Don’tpostthatfilefor

anyonetosee.Aswasmentionedalready,youkeepneededconfigurationvalues

there,butthesevaluesarealsosetasenvironmentname/valuepairsinanElastic

Beanstalkenvironment.

SharingObjects

Thenode2anddbvariablesareneededinyourroutingcode.Thedbobject

databaseconnectionwillbeusedforalloftheCRUDoperations,soyouneedto

makethat available. Youexpose these variables through middleware injection.

This means that you have a chance to inject the objects into the request

processingchainbyaddingthemaspropertiesontherequestobject.

Youplacea middleware function right atthe top of the Express chainthat

everyrequestwillhavetopassthroughfirst.Inthereyouattachnewproperties

ontherequestobjectthatisbeingpassedalong.

Asrequired, you callnext() to move the execution along to the rest of the

processingchainfortherequest.Rememberthattheusefunctionappliesacross

allrequests,sothataget,putoranyotherrequestisroutedthroughherefirst.

app.use(function(req,res,next){

req.db=db;

req.node2=node2;

next();

});

ExpressRouteHandlers

Youarealmostdonewiththemainapplicationcodethatsetseverythingup.

ThefollowingarealloftheroutesforyourHTTP/RestAPI.Thisgoesbackto

understandingwhatyourobjectsareandwhatverbseachwillsupport.Youjust

list out each object. Inside each of the supporting modules you will find the

verbsandanysub-objectsoffofthose.Thesearemodulesyouwritethatusethe

ExpressRouterobjectasexplainedinsection9.2.

//RestAPIroutes

app.use('/api/users',users);

app.use('/api/sessions',session);

app.use('/api/sharednews',sharedNews);

app.use('/api/homenews',homeNews);

Next,thereisanerrorhandlingroutethathandlesinvalidURLsthatcomein.

Thisreturnsa404codetosignalthattheresourcewasnotfound.Basically,if

noneoftheroutesmatch,thenthisonewill.Forexample,ifarequestcamein

for /api/blah, it would go here. This code activates an express error handler,

becauseitcallsnext(err).

//catcheverythingelseandforwardtoerrorhandlerasa404toreturn

app.use(function(req,res,next){

varerr=newError('NotFound');

err.status=404;

next(err);

});

Herearetheerrorhandlingroutesforwhenyouhaveanerrorreturnedinthe

code.Yougetherewhennext(err)iscalledinanyroutingcode.Thereisalsoa

handlerthatonlykicksinwhenrunninginyourdevelopmentenvironment.You

wanttodothissothatyoucanaddinwhatthestacktraceis.Thesecondhandler

istheonethatkicksin,intheproductionenvironment.

//developmenterrorhandlerthatwilladdinastacktrace

if(app.get('env')==='development'){

app.use(function(err,req,res,next){

res.status(err.status||500).json({message:err.toString(),

error:err});

console.log(err);

});

}

//productionerrorhandlerwithnostacktracesexposedtousers

app.use(function(err,req,res,next){

res.status(err.status||500).json({message:err.toString(),error:{}});

console.log(err);

});

Thefinallinesinserver.jscontainthenecessarycodethattellsExpresstobe

listeningforHTTPrequests.Inproduction,itpicksuptheportnecessarytorun

inthathostedenvironment.Thelastthreelinesshown,exporttheserverforour

testingframeworktouse.

app.set('port',process.env.PORT||3000);

varserver=app.listen(app.get('port'),function(){

console.log('Expressserverlisteningonport'+server.address().port);

});

server.db=db;

server.node2=node2;

module.exports=server;

IfyoulookatthecodeintheGitHubproject,youwillseethatIalsoadded

routehandlersforstartingandstoppingtheV8profilerandfortakingmemory

snapshots.Thesecanthenbeactivatedbyplacingacalltothatparticularroute,

andalsodeactivatedwhennotneeded.

12.6AMongoDBDocumenttoHoldNews

Stories

YouneedtocreateaMongoDBdocumenttostorethecontentsoftheshared

globalnewsstories.Thisisaone-timecreationdoneinadvancethatcanbedone

throughtheMongoDBInc.Compassapplication.Thisdocumentfunctionsasthe

holder for the master list of current news stories and top stories found on the

homepage. This is then used by all users to do matchingof their news filters

with.Thisway,eachuserdoesnotneedtofetchallthenewsindividually.This

documentwillhaveadistinctivevaluesetforthe_idpropertysoyouknowwhat

itisfor.Becarefultonotaccidentallydeleteit.Thisdocumentlooksasfollows:

{

"_id":"MASTER_STORIES_DO_NOT_DELETE",

"newsStories":[],

"homeNewsStories":[]

}

Icoveredthispreviouslyinpartoneofthisbook,soyoumighthavealready

createdthisdocument.Ifnot,youcancreateitatthistime.Thisdocumentcould

becreatedincode ifyoureally wantedto.Since itisa one-timething,Ihave

chosentouseCompasstocreateit.

12.7ACentralPlaceforConfiguration(.env)

To run your Node application locally, there will need to be certain

environmentvariablessetthatyourcodecanreference.Thesearevaluessuchas

thosetoestablishtheconnectiontotheMongoDBdatabase.Youplacean.env

fileintheprojectandthenuseannpmmoduletoconsumethat.Thistechnique

wasusedintheserver.jsfileasalreadyexplained.

Be aware that I cannot divulge the actual contents of my .env file, as you

wouldthenhaveaccesstoservicesIneedtoprotect.Replacevaluesasneededin

yourown.envfile.

//3.6mongodbdriverandlaterconnectionstring

MONGODB_CONNECT_URL=mongodb+srv://babajee:mypass@cluster0-k5ghj.mongodb.net/test

JWT_SECRET=<yoursecretkey>

NEWYORKTIMES_API_KEY=<yoursecretkey>

GLOBAL_STORIES_ID=MASTER_STORIES_DO_NOT_DELETE

MAX_SHARED_STORIES=30

MAX_COMMENTS=30

MAX_FILTERS=5

MAX_FILTER_STORIES=15

The value for the MongoDB connection URL can be found in the Atlas

portal as was explained previously. You go to the clusters page and click the

CONNECTbuttonandfindthestringforconnectingtoanapplication.Follow

theinstructionsthatarefoundthere.Itwilltellyoutoreplacetheplaceholders

foryourownpassword.

12.8HTTP/RestWebServiceAPI

ItisnowtimetofillintheRESTWebServiceAPI.TheAPIwillacceptand

pass back JSON payloads in response to HTTP requests. Eventually, you will

createtheSPAwebpagethatcallsthewebservicebeingdesignedhere.

IfyouthinkabouttheRESTAPIthatyouwanttoexpose,itbecomesclear

that you need to create all the CRUD operations for each resource that is

necessary.Yourresourcesaresessions,users,sharednewsandhomenews.

HereisatablethatlistseverythingtheRESTAPIsupports.Theidparameter

is the identifier of individual resources for a user being accessed. The sid

parameteristheidentifierforstories.

VerbandPath Result

POST

/api/sessions

Createaloginsessiontoken.

DELETE

/api/sessions/:id

Deletealoginsessiontoken.

POST

/api/users

CreateauserwiththepassedinJSONofthe

HTTPbody.

DELETE

/api/users/:id

Deleteasinglespecifieduser.

GET

/api/users/:id

ReturntheJSONofasinglespecifieduser.

PUT

/api/users/:id

Replaceauserwiththepassed-inJSONofthe

HTTPbody.

POST

/api/users/:id/savedstories

Saveastoryforuser,contentofwhichisinthe

JSONbody

DELETE

/api/users/:id/savedstories/:sid

Deleteastorythattheuserhadpreviously

saved.

POST

/api/sharednews

ShareanewsstoryascontainedintheJSON

body

GET

/api/sharednews

Getallofthesharednewsstories

DELETE

/api/sharednews/:sid

Deleteanewsstorythathadbeenshared.

POST

/api/sharednews/:sid/comments

Addacommenttoaspecifiedsharednews

story

GET

/api/homenews

Getallthehomepagenewsstories

You may have noticed that some verbs are missing that you might have

expectedtofind.Forexample,youwillnotseeaGET/api/userstogetthelistof

all users. You don’treally want other people to see everyone that is a user of

NewsWatcher,sodon’tofferthat.Youcertainlycoulddecidetoofferit,butyou

wouldthenwanttoplaceamiddlewarerestrictiononitthatonlyallowslogged

inadministratorstohaveaccesstoit.

An example of a restricted API route that you have is DELETE

/api/users/:id.Ausercanonlydeletethemselves,soyourestrictthattojustthe

logged-inuserfordeletingtheirownaccountandnotanaccountofanyoneelse.

Youcouldwritesomecodetoallowadminstobeabletodeleteanyone.

Rememberthatthetokenisusefulforrestrictingaccesswith.Itisuptoyou

todefinewhatrolesandaccessyouwillneedandthenenforceit.Inthiscase,

each call only works for that account to access that person’s own data for

whateverisauthorized.Perhapsadministratorsthatlogincanbeidentifiedand

allowedaccesstoeverything.

Visualizingthecode

Ifyourecall,thelinesbelowarefoundintheserver.jsfileandareusedtoset

upyourExpressroutehandling.ThefirsttwoExpressapplicationcallsareused

forsendingthefilesbackthattheclientbrowserapplicationwillneed.Thelast

fourExpressapplicationcallsareforroutehandlingofeverythinglistedinthe

RESTAPIresourcetable.

Herearethelinesfromtheserver.jsfileasareminder:

//MainfiletoserveupthatisbuiltbyReactbuildprocess

app.get('/',function(req,res){

res.sendFile(path.join(__dirname,'build','index.html'));

});

//ServingupofstaticcontentsuchasHTMLfortheReactSPA,images,

//CSSfiles,andJavaScriptfiles

app.use(express.static(path.join(__dirname,'build')));

//RestAPIroutes

app.use('/api/users',users);

app.use('/api/sessions',session);

app.use('/api/sharednews',sharedNews);

app.use('/api/homenews',homeNews);

Thefollowingisapictorialrepresentationofhowtheroutingcodeallhooks

together.Thisdoesnotcontainallofthefilesanddetails,butitatleastgivesyou

an idea of the routes that are being serviced. I have even included the UI

rendering React code even though that has not been covered yet. Youcan see

thatIdivideupthecodebythearchitecturallayeritresidesin.

Figure61-NewsWatcherfilediagram

YoucannowlookattheindividualfilesusedforeachoftheExpressroutes

oneatatime.Theyeachexistin theirownfile.Thesearethefileswhereyou

findtheservicingoftheindividualHTTPverbs(GET,POST,PUT,DELETE).

12.9SessionResourceRouting

(routes/session.js)

ThesessionrouteisusedintheAPItoallowpeopletologinandout.Here

arethespecificroutesforthesessionresource:

VerbandPath Result

POST

/api/sessions

Createaloginsessiontoken.

DELETE

/api/sessions/:id

Deletealoginsessiontoken.

Thepostoperationtakesauser’semailandpasswordintherequestbodyand

basically logs a user into their account. A token is sent back in the response

body.TheclientcallingcodecantaketheJWTandthenkeeppassingitbackon

subsequentcalls to identify that person. The token canbestored in client-side

storageandusedasneeded.

Thepostverb handler needs to first query for the user document to see if

theyactuallyhavearegisteredaccount.Theusermustexist,ortheycannotbe

logged in. If there is a match for the email, then the stored password hash is

validatedwithahashofthepasswordcomingin.

Ifthepasswordisvalidated,thenatokeniscreatedandpassedbackinthe

JSONpayload.Thetokenscouldbeusedforever,butitwouldbeeasytoadda

timestamp to set it to expire. With that added, you might do something like

requireanewlogineverysixmonths.

Aswasdiscussed,thereisabitofverificationthatcangoonwiththetoken.

Youcheckittomakesureitisoriginatingfromwherethetokenwasoriginally

assignedfrom.TheIPaddressandtheheadersettingforuser-agentarekeptwith

thetokenasadditionalverification.

There is no database storage of the token in the NewsWatcher app, so the

route delete does not really need to do much. There is just a simple check to

verifythatthepersonloggingoutisthesameastheonecontainedinthetoken.

Thecodeusesthejoimodule tovalidatetheincomingrequestbody object

parameters.Hereisthecodeforsession.js:

//session.js:ANode.jsModuleforsessionloginandlogout

"usestrict";

varexpress=require('express');

varbcrypt=require('bcryptjs');//Forpasswordhashcomparing

varjwt=require('jwt-simple');//Fortokenauthentication

varjoi=require('joi');//Fordatavalidation

varauthHelper=require('./authHelper');

varrouter=express.Router();

//Createasecuritytokenastheuserlogsinthatcanbepassedtothe

//clientandusedonsubsequentcalls.

//Theuseremailandpasswordaresentinthebodyoftherequest.

router.post('/',functionpostSession(req,res,next){

//Passwordmustbe7to15charactersinlengthand

//containatleastonenumericdigitandaspecialcharacter

varschema={

email:joi.string().email().min(7).max(50).required(),

password: joi.string().regex(/^(?=.*[0-9])(?=.*[!@#$%^&*])[a-zA-Z0-9!@#$%^&*]

{7,15}$/).required()

};

joi.validate(req.body,schema,function(err){

if(err)

returnnext(newError('Invalidfield:password7to15(onenumber,onespecialcharacter)'));

req.db.collection.findOne({type:'USER_TYPE',email:req.body.email},

function(err,user){

if(err)

returnnext(err);

if(!user)

returnnext(newError('Userwasnotfound.'));

bcrypt.compare(req.body.password,user.passwordHash,

functioncomparePassword(err,match){

if(match){

try{

vartoken=jwt.encode({authorized:true,sessionIP:req.ip,

sessionUA:req.headers['user-agent'],

userId:user._id.toHexString(),

displayName:user.displayName},

process.env.JWT_SECRET);

res.status(201).json({displayName:user.displayName,

userId:user._id.toHexString(),

token:token,msg:'Authorized'});

}catch(err){

returnnext(err);

}

}else{

returnnext(newError('Wrongpassword'));

}

});

//Deletethetokenasauserlogsout

router.delete('/:id',authHelper.checkAuth,function(req,res,next){

//Verifythepassedinidisthesameasthatintheauthtoken

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestforlogout'));

res.status(200).json({msg:'Loggedout'});

});

module.exports=router;

Notice the use of the middleware function authHelper.checkAuth. This is

something you will define next. This is what allows you to inject an

authorizationcheckbeforeproceedingtothefinalfunctionthatdoesthework.If

the authorization fails then the function for the end route handling will not be

called.

12.10AuthorizationTokenModule

(routes/authHelper.js)

Youhaveseenhowauserlogsinandatokenisgenerated.Younowneedto

create some middleware that will be inserted and run for verifying the token

beforeproceedingonaroutethatneedstobesecure.Thiscodewilllookatthe

passed-in token and make sure it is valid. As you know, middleware can be

insertedintoanyrouteandthatiswhatwillbehappeninghere.

Each of the routing modules will make use of the authHelper module to

verifythatavalidtokenisbeingpassedbeforeperforminganyotheraction.This

happensbecauseeachHTTP/Restcallwouldhaveanx-authheadertokenvalue

filledin.Youcoulduseaheadersuchas“Authorization:Bearer<token>”ifyou

wanttomimicstandardslikeOIDC.

YouwillseetheuseofthecheckAuthfunctioninmanyoftheroutehandlers.

Thecodeherewillsimplyverifythatthereisanx-authheaderand,ifthereis,

thendecodeitwiththesecret,thensetthedecodedobjectinarequestproperty

namedauthforfurtherusagebyanythingintheprocessingchain.Ifthetokenis

missing, has been tampered with, or does not contain what it is supposed to

contain,anerrorisreturned.

Whenyoulookatthepasswordhandlingcodeofsession.js,youseewhere

alltheinformationwasplacedinthetoken.Itreallyisuptoyoutodecidewhat

toputinthere.Withthistoken,youcanrepresenttheuserbeingsignedin.Do

not store the user password (or other sensitive data) in the token. Here is the

middlewarethatverifiesthatatokenisvalid:

//authHelper.js:ANode.jsModuletoinjectmiddlewarethatvalidatesthe

//requestheaderUsertoken.

"usestrict";

varjwt=require('jwt-simple');

//Checkforatokeninthecustomheadersettingandverifythatitis

//signedandhasnotbeentamperedwith.

//Ifnoheadertokenispresent,maybetheuser

//TheJWTSimplepackagewillthrowexceptions

module.exports.checkAuth=function(req,res,next){

if(req.headers['x-auth']){

try{

req.auth=jwt.decode(req.headers['x-auth'],process.env.JWT_SECRET);

if(req.auth&&req.auth.authorized&&req.auth.userId){

returnnext();

}else{

returnnext(newError('Userisnotloggedin.'));

}

}catch(err){

returnnext(err);

}

}else{

returnnext(newError('Userisnotloggedin.'));

}

};

IfyouprovideaUIcheck-boxfortheusertostayloggedin,thentheycan

checkthatyoucanstorethetokenonthedeviceandnothavetologthemineach

time. If a user wanted to get their token from the returned login request, they

could. If they passed it on to anyone else, their account could possibly be

compromised.Thus,theextratests.ThereisatestfortheIPaddressbeingthe

same. This may not actually work if you have users logging in from different

devicesallthetimewithdifferentIPaddresses.

12.11UserResourceRouting

(routes/users.js)

The user resource represents information for a logged in user. A user

documentretrievedbytheirIDwillcontaininformationliketheirprofilethathas

theirnewsfiltersandalsothenewsstoriesthathavematched.Foragivenuser,

youcanalsomakeRestcallstosaveastoryordeleteasavedstory.Herearethe

specificroutesfortheuserresource:

VerbandPath Result

POST

/api/users

Createauserwiththepassed-inJSONofthe

HTTPbody.

DELETE

/api/users/:id

Deleteasinglespecifieduser.

GET

/api/users/:id

ReturntheJSONofasinglespecifieduser.

PUT

/api/users/:id

Replaceauserwiththepassed-inJSONofthe

HTTPbody.

POST

/api/users/:id/savedstories

Saveastoryforuser,contentofwhichisinthe

JSONbody

DELETE

/api/users/:id/savedstories/:sid

Deleteastorythattheuserhadpreviouslysaved.

Startbylookingattherequirestatementsattheverytopofusers.js.Someof

themodulesrequiredareonesthatyouhaveseenbefore.

varexpress=require('express');

varbcrypt=require('bcryptjs');

varasync=require('async');

varjoi=require('joi');//Fordatavalidation

varauthHelper=require('./authHelper');

varObjectId=require('mongodb').ObjectID;

varrouter=express.Router();

...codecutouthere...

module.exports=router;

Let’slookattheverbhandlingfunctionsonebyone.

POST/api/users

The post verb takes a JSON payload and creates a new user account as a

documentinyourcollection.Thishappenswhenauseraccountisfirstcreated.

ApasswordispassedinaspartoftheJSONbodyandyoucreateahashofitto

storeintheMongoDBdatabasedocumentforthatuser.

This call will fail if there already exists a document in the collection with

thatemailvaluealready.Relyingonanemailaddresstoidentifyauseraccount

is one way to keep user accounts unique and identifiable. The mongodb

findOne()functioniswhatisusedtoseeifauseraccountexistedalready.

The code goes ahead and creates all of the properties ever needed in the

document.Defaultvaluesareusedthatmakesense.Thereisevenasamplenews

filtersetupfortheuser.

Noticethecalltonode2.send().Thisisusedtopassamessagetotheforked

Nodeprocesstooffloadthefilterprocessingforthatnewlyaddeduser.Thiscall

willtakethenewaccountanddotheinitialmatchingofstoriesforthefilter.I

havenotshownyouthecodefortheforkedprocessyet,butyoucanseethatthe

conceptisverysimple.

Whathappensrightatthestart,isthevalidationofthepassed-inJSONbody.

Youwant tomakesure thatthereare noextraproperties, andvalidatethat the

allowedonesconformtosomeknowntypesandhavesafevalues.ThejoiNPM

module is used to perform the validations. Here is the code for the post verb

handler:

router.post('/',functionpostUser(req,res,next){

//Passwordmustbe7to15charactersinlengthandcontainatleastone

//numericdigitandaspecialcharacter

varschema={

displayName:joi.string().alphanum().min(3).max(50).required(),

email:joi.string().email().min(7).max(50).required(),

password: joi.string().regex(/^(?=.*[0-9])(?=.*[!@#$%^&*])[a-zA-Z0-9!@#$%^&*]

{7,15}$/).required()

};

joi.validate(req.body,schema,function(err,value){

if(err)

returnnext(newError('Invalidfield:displayname3to50alphanumeric,validemail,password7to

15(onenumber,onespecialcharacter)'));

req.db.collection.findOne({type:'USER_TYPE',email:req.body.email},function(err,doc){

if(err)

returnnext(err);

if(doc)

returnnext(newError('Emailaccountalreadyregistered'));

varxferUser={

type:'USER_TYPE',

displayName:req.body.displayName,

email:req.body.email,

passwordHash:null,

date:Date.now(),

completed:false,

settings:{

requireWIFI:true,

enableAlerts:false

newsFilters:[{

name:'TechnologyCompanies',

keyWords:['Apple','Microsoft','IBM','Amazon','Google','Intel'],

enableAlert:false,

alertFrequency:0,

enableAutoDelete:false,

deleteTime:0,

timeOfLastScan:0,

newsStories:[]

}],

savedStories:[]

};

bcrypt.hash(req.body.password,10,functiongetHash(err,hash){

if(err)

returnnext(err);

xferUser.passwordHash=hash;

req.db.collection.insertOne(xferUser,functioncreateUser(err,result){

if(err)

returnnext(err);

req.node2.send({msg:'REFRESH_STORIES',doc:result.ops[0]});

res.status(201).json(result.ops[0]);

});

Users are allowed to sign up for an account by having them provide their

username, email, and a password. You will not store the actual password, but

will instead store an encrypted hashed value of the password. Even if anyone

weretogetaholdofthatforauser,theywouldstillnotbeabletologinwithit

asitisextremelydifficulttodecryptthatintothepassword.

Onceapersonisregisteredasauser,thenthe/api/session/pathwillbeused

toaccepttheiremailandpasswordtogettheirsessiongoingeachtimetheywant

tologinanduseNewsWatcher.

Asexplainedinthechapteronauthenticationandauthorization,youwillbe

sendingatokenintheheaderofeachHTTP/Restrequest.Whenauserlogsin,

they get a token that is subsequently used to identify them for all further

interactions.

DELETE/api/users/:id

With this path, you see the use of a passed in id. It comes in through the

mechanismofExpress.SpecifyingthepathlikethiswillhaveExpresscreatea

propertyofreq.params.id.Whatyouneedtodoistoverifythattherequestfora

deletion of a user is actually the id that exists in the token. This way a user

cannotdeleteanaccountthatdoesnotbelongtothem.

Lookatsession.jsagainandyouseewherethemongodb_idpropertyofthe

retrieveddocumentiscaptured. Thisiswhat isgoing tobepassed backinthe

RestrequestURLpathportiontoidentifyauser.

The middleware function authHelper.checkAuth is injected to do the

verification that a valid token exists for the request. That middleware-injected

function will return an error if the token is not acceptable and then the route

functionwillnevergetcalled.

If everything proceeds correctly, the route function executes, and the

document is removed from your collection, and the user account is gone.

findOneAndDelete()isusedastherewouldbeoneandonlyonedocumentwith

that_id.

Thereisahelperfunctionfromthemongodbmoduletotakethestringand

getitintotheproperformneeded.Hereisthecodefortheuserdeletionhandler:

router.delete('/:id',authHelper.checkAuth,function(req,res,next){

//Verifythatthepassedinidtodeleteisthesameasthatinthe

//authtoken

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestforaccountdeletion'));

//MongoDBshoulddotheworkofqueuingthisupandretryingifthereis

//aconflict,Accordingtotheirdocumentation.

//Thisrequiresawritelockontheirpart.

req.db.collection.findOneAndDelete(

{type:'USER_TYPE',_id:ObjectId(req.auth.userId)},

function(err,result){

if(err){

console.log("POSSIBLEUSERDELETIONCONTENTION?err:",err);

returnnext(err);

}elseif(result.ok!=1){

console.log("POSSIBLEUSERDELETIONERROR?result:",result);

returnnext(newError('Accountdeletionfailure'));

}

res.status(200).json({msg:"UserDeleted"});

});

GET/api/users/:id

This route handler retrieves a single user by their id. The app would have

alreadycalledtogetasessiontokenandthenhaveaccesstotheidoftheuserto

passitintothisAPIcalltoretrievetheuserdocument.Sinceyouactuallyhave

the_id(objectid),youcanretrievethedocumentfasterthanifyouhadtoquery

foritsomeotherway.Thereisalwaysanindexcreatedforthe_idproperty.

Theretrievalisdoneandpopulatedwithatransferobject.Noticethatyouare

alsotweakingthe HTTPheaderfor theresponse. Thatisnecessary inorder to

stop caching from happening. Otherwise, when you get to the UI presentation

codeandaretryingtoretrieveauser,youmightnotgetthemostup-to-dateone.

Hereisthecodefortheroutehandlertogetasingleuserbytheirid:

router.get('/:id',authHelper.checkAuth,function(req,res,next){

//Verifythatthepassedinidtodeleteisthesametheauthtoken

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestforaccountfetch'));

req.db.collection.findOne({type:'USER_TYPE',

_id:ObjectId(req.auth.userId)},

function(err,doc){

if(err)returnnext(err);

varxferProfile={

email:doc.email,

displayName:doc.displayName,

date:doc.date,

settings:doc.settings,

newsFilters:doc.newsFilters,

savedStories:doc.savedStories

};

res.header("Cache-Control","no-cache,no-store,must-revalidate");

res.header("Pragma","no-cache");

res.header("Expires",0);

res.status(200).json(xferProfile);

});

PUT/api/users/:id

Aputisusedtoupdateauser,suchasinthecasewheretheyhavealtered

thiernewsfilters.Thecodeisverysimilartowhatyouneededfortheinitialpost

of the user, except you now need to worry about a conflict happening upon a

databasewriteoperation.Hereisthecodefortheuserupdatehandler:

router.put('/:id',authHelper.checkAuth,function(req,res,next){

//Verifythatthepassedinidisthesameasthatintheauthtoken

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestforaccountdeletion'));

//LimitthenumberofnewsFilters

if(req.body.newsFilters.length>process.env.MAX_FILTERS)

returnnext(newError('ToomanynewsnewsFilters'));

//clearoutleadingandtrailingspaces

for(vari=0;i<req.body.newsFilters.length;i++){

if("keyWords"inreq.body.newsFilters[i]&&

req.body.newsFilters[i].keyWords[0]!=""){

for(varj=0;j<req.body.newsFilters[i].keyWords.length;j++){

req.body.newsFilters[i].keyWords[j]=

req.body.newsFilters[i].keyWords[j].trim();

}

//ValidatethenewsFilters

varschema={

name:joi.string().min(1).max(30).regex(/^[-_a-zA-Z0-9]+$/).required(),

keyWords:joi.array().max(10).items(joi.string().max(20)).required(),

enableAlert:joi.boolean(),

alertFrequency:joi.number().min(0),

enableAutoDelete:joi.boolean(),

deleteTime:joi.date(),

timeOfLastScan:joi.date(),

newsStories:joi.array(),

keywordsStr:joi.string().min(1).max(100)

};

async.eachSeries(req.body.newsFilters,function(filter,innercallback){

joi.validate(filter,schema,function(err){

innercallback(err);

});

},function(err){

if(err){

returnnext(err);

}else{

//MongoDBimplementsoptimisticconcurrencyforus.

//Wewerenotholdingontothedocumentanyway,sowejustdoa

//quickreadandreplaceofjustthosepropertiesandnotthe

//completedocument.

//Itmattersifnewsstorieswereupdatedinthemeantime(i.e.

//usersattheretakingtheirtimeupdatingtheirnewsprofile)

//becausewewillforcethattoupdateaspartofthisoperation.

//Weneedthe{returnOriginal:false},soatestcouldverifywhat

//happened,otherwisethedefaultistoreturntheoriginal.

req.db.collection.findOneAndUpdate({type:'USER_TYPE',

_id:ObjectId(req.auth.userId)},

{$set:{settings:{requireWIFI:req.body.requireWIFI,

enableAlerts:req.body.enableAlerts},

newsFilters:req.body.newsFilters}},

{returnOriginal:false},

function(err,result){

if(err){

console.log("+++POSSIBLEUSERPUTCONTENTIONERROR?+++err:",

err);

returnnext(err);

}elseif(result.ok!=1){

console.log("+++POSSIBLECONTENTIONERROR?+++result:",

result);

returnnext(newError('UserPUTfailure'));

}

req.node2.send({msg:'REFRESH_STORIES',doc:result.value});

res.status(200).json(result.value);

});

}

});

Noticethecodetolimitthenewsfiltersize.YouwillputcodeintheUIto

limit that as well, but that could be hacked in the browser, or by someone

sendingabogusputrequest.You have to guard against potential tampering as

youwouldotherwisehaveacrash,oratleastafailureoftheMongoDBupdate.

Thereistheuseofthis amazinglyusefulmodulecalledasync.This allows

calls to the joi library to happen over and over and allows waiting for each

callbacktoreturnforeachofthefiltersforauser.Whenallareprocessed,the

finalanonymousfunctioniscalled.

The$setoperation is usedto update only individualproperties and notthe

entiredocument.

Thereissomeerrorcheckingcodeintheretodetectanycontentionerror.I

hadtriedoverandoverandhaveneverseenonehappen.

POST/api/users/:id/savedstories

In the user document, there is an array used for saving stories that a user

wants to keep around. This route will take the JSON of the passed-in request

body as the story to save. The id in the route is the id of the user that is

requestingthesavingofthestory.

Thereareafewchecksthatneedtohappenbeforesavingastory.Youneed

toverifythatthestoryisnotalreadyinserted.Thereisalsoalimitonthenumber

ofstoriesthatcanbesaved,sothathastobecheckedalso.

Storieshaveanidassociatedwiththemtobeabletoidentifythemincases

likethiswhereyoudon’twantduplicatessavedorshared.Youwilllaterseethe

codethatcreatesthatid.Hereisthecodeforpostingastorytobesaved:

router.post('/:id/savedstories',authHelper.checkAuth,function(req,res,

next){

//Verifythattheidtodeleteisthesameasintheauthtoken

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestforsavingstory'));

//Validatethebody

varschema={

contentSnippet:joi.string().max(200).required(),

date:joi.date().required(),

hours:joi.string().max(20),

imageUrl:joi.string().max(300).required(),

keep:joi.boolean().required(),

link:joi.string().max(300).required(),

source:joi.string().max(50).required(),

storyID:joi.string().max(100).required(),

title:joi.string().max(200).required()

};

joi.validate(req.body,schema,function(err){

if(err)

returnnext(err);

//makesure:

//A.Storyisnotalreadyinthere.

//B.Welimitthenumberofsavedstoriesto30

//Notallowedatfreetier!!!

//req.db.collection.findOneAndUpdate({type:'USER_TYPE',

//_id:ObjectId(req.auth.userId),

//$where:'this.savedStories.length<29'},

req.db.collection.findOneAndUpdate({type:'USER_TYPE',

_id:ObjectId(req.auth.userId)},

{$addToSet:{savedStories:req.body}},

{returnOriginal:true},

function(err,result){

if(result&&result.value==null){

returnnext(newError('Overthesavelimit,

orstoryalreadysaved'));

}elseif(err){

console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);

returnnext(err);

}elseif(result.ok!=1){

console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);

returnnext(newError('Storysavefailure'));

}

res.status(200).json(result.value);

});

Note:The$whereabovedoesnotworkwiththefreetierofAtlasandmustbe

omitted.

DELETE/api/users/:id/savedstories/:sid

Thisissimilartotheotherfunctionsandaccomplishestheverificationofthe

storyexisting,beforebeingabletodeleteit.The$pulloperatorisusedwiththe

arraypropertyofthedocumenttodeletetheentry.Hereisthecodefordeletinga

savedstory:

router.delete('/:id/savedstories/:sid',authHelper.checkAuth,function(req,res,next){

if(req.params.id!=req.auth.userId)

returnnext(newError('Invalidrequestfordeletionofsavedstory'));

req.db.collection.findOneAndUpdate({type:'USER_TYPE',

_id:ObjectId(req.auth.userId)},

{$pull:{savedStories:{storyID:req.params.sid}}},

{returnOriginal:true},

function(err,result){

if(err){

console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);

returnnext(err);

}elseif(result.ok!=1){

console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);

returnnext(newError('Storydeletefailure'));

}

res.status(200).json(result.value);

});

12.12HomeNewsRouting

(routes/homeNews.js)

HomenewsstoriesarethosestoriesthatarevisiblewhentheapplicationUI

isfirstseenbytheuser.ThereisonlyonesinglerouteneededforthehomeNews

resource.Auserdoesnotneedtobeloggedintoseethesestories.

VerbandPath Result

GET

/api/homenews

Getallofthetopnewsstories

Atthetopandbottomofthefileistheusualcodetoexposethemoduleas

shownhere:

"usestrict";

varexpress=require('express');

varrouter=express.Router();

...thispartleftout...

module.exports=router;

Hereistheroutehandlerforsettingthenewsstoriesforthehomepage.

GET/api/homenews

Retrievingalltopnewsstoriesisdonebydirectlygettingthearraythatholds

themfromouronesingledocumentforthatpurpose.Theyarethesameforall

users.

router.get('/',function(req,res,next){

req.db.collection.findOne({ _id: process.env.GLOBAL_STORIES_ID }, { homeNewsStories: 1 },

function(err,doc){

if(err)

returnnext(err);

res.status(200).json(doc.homeNewsStories);

});

The array also works great, as you can send that back in the response and

thenReactcanbinditontheclientside.

12.13SharedNewsRouting

(routes/sharedNews.js)

Shared news stories are those that are seen by all users. People can save,

view and comment on news stories. Here are the specific routes for the

sharedNewsresource:

VerbandPath Result

POST

/api/sharednews

ShareanewsstoryascontainedintheJSON

body

GET

/api/sharednews

Getallofthesharesnewsstories

DELETE

/api/sharednews/:sid

Deleteanewsstorythathasbeenshared.

POST

/api/sharednews/:sid/comments

Addacommenttoaspecifiedsharednews

story

Atthetopandbottomofthefileistheusualcodeasshownhere:

"usestrict";

varexpress=require('express');

varjoi=require('joi');//Fordatavalidation

varauthHelper=require('./authHelper');

varrouter=express.Router();

...thispartleftout...

module.exports=router;

Hereareeachoftheroutepathhandlers.

POST/api/sharednews

Thiscodeisverysimilartothecodeyoualreadysawforsavingastoryin

user.js.Theonlydifferenceisthatnow,thestoryisbeingcopiedintoadifferent

documentwhereallNewsWatcheruserscanviewstoriesandcommentonthem.

Thereisalimitsetforthenumberofpossiblesharedstories.Thereisalsoa

test to make sure the story was not already shared. If all looks good, the

documentiscreated.Hereisthecodeforsharingastory:

router.post('/',authHelper.checkAuth,function(req,res,next){

//Validatethebody

varschema={

contentSnippet:joi.string().max(200).required(),

date:joi.date().required(),

hours:joi.string().max(20),

imageUrl:joi.string().max(300).required(),

keep:joi.boolean().required(),

link:joi.string().max(300).required(),

source:joi.string().max(50).required(),

storyID:joi.string().max(100).required(),

title:joi.string().max(200).required()

};

joi.validate(req.body,schema,function(err){

if(err)returnnext(err);

//Wefirstmakesurewearenotatthecountlimit.

req.db.collection.count({type:'SHAREDSTORY_TYPE'},

function(err,count)

{

if(err)returnnext(err);

if(count>process.env.MAX_SHARED_STORIES)

returnnext(newError('Sharedstorylimitreached'));

//Makesurethestorywasnotalreadyshared

req.db.collection.count({type:'SHAREDSTORY_TYPE',

_id:req.body.storyID},function(err,count)

{

if(err)returnnext(err);

if(count>0)

returnnext(newError('Storywasalreadyshared.'));

//Wesettheidandguaranteeuniquenessorfailurehappens

varxferStory={

_id:req.body.storyID,

type:'SHAREDSTORY_TYPE',

story:req.body,

comments:[{

displayName:req.auth.displayName,

userId:req.auth.userId,

dateTime:Date.now(),

comment:req.auth.displayName+

"thoughteveryonemightenjoythis!"

}]

};

req.db.collection.insertOne(xferStory,

functioncreateUser(err,result)

{

if(err)

returnnext(err);

res.status(201).json(result.ops[0]);

});

GET/api/sharednews

Retrievingallsharedstoriesisdonebydirectlygettingthedocumentsoftype

SHAREDSTORY_TYPEasfollows:

router.get('/',authHelper.checkAuth,function(req,res,next){

req.db.collection.find({type:'SHAREDSTORY_TYPE'}).toArray(function(err,docs){

if(err)

returnnext(err);

res.status(200).json(docs);

});

Youknowtherewillnotbemorethan30soitisoktohaveanarrayreturned

andnotuseacursortoiteratethroughtheresults.Thearraytypeworksgreat,as

youcansendthatbackintheresponseandthenReactcanbinditontheclient

side.

DELETE/api/sharednews/:sid

Individualsharedstoriescanbedeleted.Youwillnotactuallybecallingthis

fromthepresentationlayer,butneeditjustfortestingpurposestocleanupafter

yourself.Itcaneitherbecommentedoutorhavesomechecksdonetoonlyallow

anadminaccounttocallit.Thecodeisasfollows:

router.delete('/:sid',authHelper.checkAuth,function(req,res,next){

req.db.collection.findOneAndDelete({type:'SHAREDSTORY_TYPE',

_id:req.params.sid},function(err,result)

{

if(err){

console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);

returnnext(err);

}elseif(result.ok!=1){

console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);

returnnext(newError('Sharedstorydeletionfailure'));

}

res.status(200).json({msg:"SharedstoryDeleted"});

});

POST/api/sharednews/:sid/comments

Toaddacomment,youneedtheidofthestoryandtheJSONbodyforthe

comment. Since you have a partially normalized design here with separate

documentsforeachstory,youwillnothaveasmuchwritecontentionthatway.

Therewillstillbeconcurrentaccessissuesforeachindividualstoryasmultiple

commentadditionswillpossiblyconflict.ThisiswhythefindOneAndUpdate()

callisused,asitwillhandlethisforyou.

Noticethatthiscanfailifthere arealready30commentsadded.Thereare

threedifferentpartstothequerycriteriaused.Thefirsttwonarrowitdownto

exactly what is being searched for. Then the $where operator is used and the

actualJavaScriptobjectisaccessedtocheckthearraylength.Thesharedstory

commentisaddedasfollows:

router.post('/:sid/Comments',authHelper.checkAuth,function(req,res,next){

//Validatethebody

varschema={

comment:joi.string().max(250).required()

};

joi.validate(req.body,schema,function(err){

if(err)

returnnext(err);

varxferComment={

displayName:req.auth.displayName,

userId:req.auth.userId,

dateTime:Date.now(),

comment:req.body.comment.substring(0,250)

};

//Notallowedatfreetier!!!req.db.collection.findOneAndUpdate({

//type:'SHAREDSTORY_TYPE',_id:req.params.sid,$where:

//'this.comments.length<29'},

req.db.collection.findOneAndUpdate({type:'SHAREDSTORY_TYPE',

_id:req.params.sid},

{$push:{comments:xferComment}},

function(err,result){

if(result&&result.value==null){

returnnext(newError('Commentlimitreached'));

}elseif(err){

console.log("+++POSSIBLECONTENTIONERROR?+++err:",err);

returnnext(err);

}elseif(result.ok!=1){

console.log("+++POSSIBLECONTENTIONERROR?+++result:",result);

returnnext(newError('Commentsavefailure'));

}

res.status(201).json({msg:"Commentadded"});

});

Note:The$whereabovedoesnotworkwiththefreetierofAtlasandmustbe

omitted.

12.14ForkedNodeProcess

(app_FORK.js)

Younever want to have any compute-intensive code running in your main

Node.js process. If you do, it will overwhelm the V8 JavaScript processing

thread and your web service will become unresponsive. There are reasonable

solutionstothis,suchasforkingoffotherprocessesfromyourmainprocessand

having code execute there. Remember also, that I said it would probably be

bettertouseAWSLambdaforthistypeofprocessing.

In order to architect your application correctly, you need to consider what

needstobemovedofftothesecondaryprocesses.InthecaseofNewsWatcher,

youcanidentifyafewpiecesofcodethatreallyneedtobesentofftoberunon

asecondNode.jsprocessthatiswaitingtodoanyprocessing.

Youcancreateafilenamedapp_FORK.jsandputafewfunctionsofcodein

there.Onesectionofcodewouldbethatwhichisperiodicallyrunonatimerfor

any batch type of work. For example, you need to populate the master news

document with the latest news stories every once in a while and then run

somethingtomatchallofthefiltersoftheusers.

Other code in app_Fork.js signaled to run by sending a message to this

secondNodeprocessfromthemainprocess.Forexample,ifausereveralters

theirfilters,coderunstoupdatethestoriesthatmatchforthatsingleuser.

Let’sstartwiththetopoftheapp_FORK.jsfileandlookattheinitialization

code. The first lines will set up what is required for your module usage. One

thingtonoteisthatthedatabaseconnectioncannotbesharedacrossprocesses,

soyouneedtoestablishthatagainhere.

"usestrict";

varbcrypt=require('bcryptjs');

varhttps=require("https");

varasync=require('async');

varassert=require('assert');

varObjectId=require('mongodb').ObjectID;

varMongoClient=require('mongodb').MongoClient;

varglobalNewsDoc;

constNEWYORKTIMES_CATEGORIES=["home","world","national","business","technology"];

//MongoDBdatabaseconnectioninitialization

vardb={};

MongoClient.connect(process.env.MONGODB_CONNECT_URL,function(err,client){

assert.equal(null,err);

db.client=client;

db.collection=client.db('newswatcherdb').collection('newswatcher');

console.log("ForkisconnectedtoMongoDBserver");

});

Let’slookathowyoucommunicatebackandforthbetweenNodeprocesses.

ThereisaglobalvariablemadeavailableinNodenamedprocess.Itisusedfor

accessing process-related properties and functions. One of those functions is

send().Itcanbeusedtosendmessagesbacktotheparentprocessthatforkedthe

process.Theon()functionisforhandlingmessagessenttothisforkedprocess

fromthemainprocess.

process.on('message',function(m){

if(m.msg){

if(m.msg=='REFRESH_STORIES'){

setImmediate(function(doc){

refreshStoriesMSG(doc,null);

},m.doc);

}

}else{

console.log('Messagefrommaster:',m);

}

});

Theonemessagesentfromthemainprocessistohandlechangestoauser’s

filter.Youschedulethehandlingofthatandreturnimmediatelyfromtheevent.

It needs to be delayed, or queued up as you could have other scheduled timer

functions firing off, as well as other message requests coming from different

users. The setImmediate()function is used to set up callbacks to run after I/O

handlinginthissecondnodeprocess.

Refreshofauser’sfilters

Youcannowlookatthefunctionthatreactstoauserwhohasjustupdated

their filters. This would be the code that is run in response to your message

handling.Thebasicalgorithmloopsthrougheachfilterofauser.Foreachfilter,

the code checks if there are any stories in the master news list that match the

keywords.Thereisalimittothenumberofmatchingstories.

Whentheupdateoftheuserdocumenthappens,the$setoperatorisusedand

onlya singleproperty of thedocument isupdated. Just thearray propertythat

holdsthenewsfilters.

Thereiscodeinherefortestingpurposes.Thisisusedtobeabletoverify

youhaveaknownnewsstorytotestforaspecialpredeterminedkeywordstring.

Youwilllaterseehowthisisusedwhenthetestcodeisdiscussed.Hereisthe

codeforuserstorymatching:

functionrefreshStoriesMSG(doc,callback){

if(!globalNewsDoc){

db.collection.findOne({_id:process.env.GLOBAL_STORIES_ID},

function(err,gDoc)

{

if(err){

console.log('FORK_ERROR:readDocument()readerr:'+err);

if(callback)

returncallback(err);

else

return;

}else{

globalNewsDoc=gDoc;

refreshStories(doc,callback);

}

});

}else{

refreshStories(doc,callback);

}

functionrefreshStories(doc,callback){

//LoopthroughallnewsFiltersandseekmatchesforallreturnedstories

for(varfilterIdx=0;filterIdx<doc.newsFilters.length;filterIdx++)

{

doc.newsFilters[filterIdx].newsStories=[];

for(vari=0;i<globalNewsDoc.newsStories.length;i++){

globalNewsDoc.newsStories[i].keep=false;

}

//IftherearekeyWords,thenfilterbythem

if("keyWords"indoc.newsFilters[filterIdx]&&

doc.newsFilters[filterIdx].keyWords[0]!=""){

varstoriesMatched=0;

for(vari=0;i<doc.newsFilters[filterIdx].keyWords.length;i++){

for(varj=0;j<globalNewsDoc.newsStories.length;j++){

if(globalNewsDoc.newsStories[j].keep==false){

vars1=globalNewsDoc.newsStories[j].title.toLowerCase();

vars2=

globalNewsDoc.newsStories[j].contentSnippet.toLowerCase();

varkeyword=

doc.newsFilters[filterIdx].keyWords[i].toLowerCase();

if(s1.indexOf(keyword)>=0||s2.indexOf(keyword)>=0){

globalNewsDoc.newsStories[j].keep=true;

storiesMatched++;

}

if(storiesMatched==process.env.MAX_FILTER_STORIES)

break;

}

if(storiesMatched==process.env.MAX_FILTER_STORIES)

break;

}

for(vark=0;k<globalNewsDoc.newsStories.length;k++){

if(globalNewsDoc.newsStories[k].keep==true){

doc.newsFilters[filterIdx].newsStories.

push(globalNewsDoc.newsStories[k]);

}

//Forthetestruns,wecaninjectnewsstoriesunderourcontrol.

if(doc.newsFilters.length==1&&

doc.newsFilters[0].keyWords.length==1

&&doc.newsFilters[0].keyWords[0]=="testingKeyword"){

for(vari=0;i<5;i++){

doc.newsFilters[0].newsStories.push(globalNewsDoc.newsStories[0]);

doc.newsFilters[0].newsStories[0].title="testingKeywordtitle"+i;

}

//Dothereplacementofthenewsstories

db.collection.findOneAndUpdate({_id:ObjectId(doc._id)},{$set:{"newsFilters":doc.newsFilters}

},function(err,result){

if(err){

console.log('FORK_ERRORReplaceofnewsStoriesfailed:',err);

}elseif(result.ok!=1){

console.log('FORK_ERRORReplaceofnewsStoriesfailed:',result);

}else{

if(doc.newsFilters.length>0){

console.log({ msg: 'MASTERNEWS_UPDATE first filter news length = ' +

doc.newsFilters[0].newsStories.length});

}else{

console.log({msg:'MASTERNEWS_UPDATEnonewsFilters'});

}

if(callback)

returncallback(err);

});

}

Thissecondfunctionisusedinmultipleplaces,soIpulleditouttobeonits

own.

Timereventtopopulatethemasternewslist

Everyfewhours,afunctionrunsthatfetchesallnewsstoriesfromthesource

newsserviceprovider.Thismeansthatitworksonabatchofdataandcouldtake

sometimebeforeitfinishes.ThisfunctionwasplacedinNodeprocessbecause

itislong-running.Itisstillsignificantenoughthatitisbettertoplaceitthereso

astonotupsetthemainNodeprocessCPUusage.Thiscouldbeoffloadedtoa

completelydifferentmachinethroughsomeotherqueuingmechanism.

Thefirstthingthathappenswhenthetimeintervalfires,isthesendingifan

HTTP request to the news feed API provided by the New York Times API

service. The results from that are placed into formatted news elements in the

master document newsStories array. Notice the use of the async module to be

abletoloopanumberoftimesandalsoseta.5seconddelaybetweeneachbatch

newsrequestfromNYT.Thisisbecausethereisarestrictionwiththeusagesuch

thatyoucan’tcallitmorethanfivetimesasecond,ortheymaydisableyourIP

addressfromaccessingtheirAPI.

Thereisanidneededforeachstory.AGUIDcouldhavebeengenerated,but

theproblemisthatthesamestorymightappearagaininthenextbatchofnews

andthatwouldcauseproblemsifyouthoughtitwasanewnewsstory.Ahashof

the link willturn out to be the best way to uniquely identify a story. The link

itselfwouldbeunusablelaterinaURLtopassinasanID,butthehashvalue

works,aslongasyoureplacecertaincharactersofit.

Onceallofthenewsstoriesareinplace,youcangothroughalloftheuser

documents and, for each user, do the story matching against the news filters.

This is done in a somewhat tricky way. The async.doWhilst() functionality is

used.Thisway,itcanhandlethedifficultyofmanagingmultipleasynccalls,one

at a time, in a simple way. The code will keep running as long as there is

processingtodo.

Youhave to consider the throughput capability on the MongoDB side and

notoverwhelm it,so itis goodto have thisprocessing serialized.Youneed to

keepplentyofheadroomforyournormaluserinteractions.Hereisthecode:

varcount=0;

newsPullBackgroundTimer=setInterval(function(){

//TheNewYorkTimesnewsservicecan’tbecalledcalledmorethanfive

//timesasecond.Wecallitoverandoveragain,becausethereare

//multiplenewscategoryis,sospaceeachoutbyhalfasecond

vardate=newDate();

console.log("app_FORK:datetimetick:"+date.toUTCString());

async.timesSeries(NEWYORKTIMES_CATEGORIES.length,function(n,next){

setTimeout(function(){

console.log('GetnewsstoriesfromNYT.Pass#',n);

try{

https.get({

host:'api.nytimes.com',

path:'/svc/topstories/v2/'+NEWYORKTIMES_CATEGORIES[n]+

'.json',

headers:{'api-key':process.env.NEWYORKTIMES_API_KEY}

},function(res){

varbody='';

res.on('data',function(d){

body+=d;

});

res.on('end',function(){

next(null,body);

});

}).on('error',function(err){

//handleerrorswiththerequestitself

console.log({msg:'FORK_ERROR',Error:err.message});

return;

});

}

catch(err){

count++;

if(count==3){

console.log('app_FORK.js:shutingdowntimer:'+err);

clearInterval(newsPullBackgroundTimer);

clearInterval(staleStoryDeleteBackgroundTimer);

process.disconnect();

}

else{

console.log('app_FORK.jserror.err:'+err);

}

},500);

},function(err,results){

if(err){

console.log('failure');

}else{

console.log('success');

//Dothereplacementofthenewsstoriesinthesinglemasterdoc

db.collection.findOne({_id:process.env.GLOBAL_STORIES_ID},

function(err,gDoc){

if(err){

console.log({msg:'FORK_ERROR',Error:'Errorwiththeglobalnewsdocreadrequest:'+

JSON.stringify(err.body,null,4)});

}else{

gDoc.newsStories=[];

gDoc.homeNewsStories=[];

varallNews=[];

for(vari=0;i<results.length;i++){

try{

varnews=JSON.parse(results[i]);

}catch(e){

console.error(e);

return;

}

for(varj=0;j<news.results.length;j++){

varxferNewsStory={

link:news.results[j].url,

title:news.results[j].title,

contentSnippet:news.results[j].abstract,

source:news.results[j].section,

date:newDate(news.results[j].updated_date).getTime()

};

//Onlytakestorieswithimages

if(news.results[j].multimedia.length>0){

xferNewsStory.imageUrl=news.results[j].multimedia[0].url;

allNews.push(xferNewsStory);

//Populatethehomepagestories

if(i==0){

gDoc.homeNewsStories.push(xferNewsStory);

}

async.eachSeries(allNews,function(story,innercallback){

bcrypt.hash(story.link,10,functiongetHash(err,hash){

if(err)

innercallback(err);

//Onlyaddthestoryifitisnotintherealready.

//StoriesonNYTcanbesharedbetweencategories

story.storyID=hash.replace(/\+/g,'-').

replace(/\//g,'_').replace(/=+$/,'');

if(gDoc.newsStories.findIndex(function(o){

if(o.storyID==story.storyID||o.title==story.title)

returntrue;

else

returnfalse;

})==-1){

gDoc.newsStories.push(story);

}

innercallback();

});

},function(err){

if(err){

console.log('failureonstoryidcreation');

}else{

console.log('storyidcreationsuccess');

globalNewsDoc=gDoc;

setImmediate(function(){

refreshAllUserStories();

});

}

});

}

});

}

});

},240*60*1000);

functionrefreshAllUserStories(){

db.collection.findOneAndUpdate({_id:globalNewsDoc._id},

{$set:{newsStories:globalNewsDoc.newsStories,

homeNewsStories:globalNewsDoc.homeNewsStories}},

function(err,result)

{

if(err){

console.log('FORK_ERRORReplaceofglobalnewsStoriesfailed:',err);

}elseif(result.ok!=1){

console.log('ReplaceofglobalnewsStoriesfailed:',result);

}else{

//ForeachNewsWatcheruser,donewsmatchingontheirnewsFilters

varcursor=db.collection.find({type:'USER_TYPE'});

varkeepProcessing=true;

async.doWhilst(

function(callback){

cursor.next(function(err,doc){

if(doc){

refreshStories(doc,function(err){

callback(null);

});

}else{

keepProcessing=false;

callback(null);

}

});

function(){returnkeepProcessing;},

function(err){

console.log('Timer:Refreshedandmatched.err:',err);

});

}

});

}

Note:Anytimeyouusetheseasync functions,youmustbereallycarefulof

howtheyoperatewiththeirasyncandsynccapabilitiesandmakesureyoucall

therequiredcallbackscorrectlyintherightplace.Errorhandlingcanalsobea

bittrickyhere.

Deletingoldstories

Thereneedstobecodethatwilldeletesharedstoriesafteracertainamount

of time. This becomes another timer that goes off periodically to do this

processing.Thecodeisasfollows:

staleStoryDeleteBackgroundTimer=setInterval(function(){

db.collection.find({type:'SHAREDSTORY_TYPE'}).toArray(

function(err,docs){

if(err){

console.log('Forkcouldnotgetsharedstories.err:',err);

return;

}

async.eachSeries(docs,function(story,innercallback){

//Gooffthedateofthetimethestorywasshared

vard1=story.comments[0].dateTime;

vard2=Date.now();

vardiff=Math.floor((d2-d1)/3600000);

if(diff>72){

db.collection.findOneAndDelete({type:'SHAREDSTORY_TYPE',

_id:story._id},

function(err,result){

innercallback(err);

});

}else{

innercallback();

}

},function(err){

if(err){

console.log('stalestorydeletionfailure');

}else{

console.log('stalestorydeletionsuccess');

}

});

},24*60*60*1000);

12.15SecuringwithHTTPS

Atthispoint,Ineedtodiscussanimportantsecuritymeasurethatneedstobe

putintoplace.Youjustcan’thostyourRESTAPIendpointwithoutencrypting

trafficbackandforth.YoufixthisbyonlypermittingHTTPSconnections.That

wayalltrafficisencryptedandsignedsoastobetamper-resistantandharderto

eavesdropon.

SincetheNode.jsserviceisexposedthroughtheElasticBeanstalkapp,you

don’tneedtomakeanychangestoyourNode.jscodetosecureit,itisalldone

throughAWS.IfyourNode.jsinstancewasexposeddirectlytotheInternetand

servingupthetrafficdirectly,thenyouwoulddosomesimpleconfigurationon

the Node.js side to install a certificate and key files and then make a few

modificationstotheNodecode.

Inthiscase,ElasticBeanstalkkeepsNodefrombeingdirectlyexposed.The

ElasticBeanstalkserviceactsasareverseproxyandthatiswhereyouneedto

setupSSL.Youwillneedtogetyourowndomainnameandinstallacertificate

foryourElasticBeanstalkservice.

Note:WhenyoulaunchyourapplicationthroughVSCode,itwillnotaccept

HTTPS locally on your machine. However, tests run against a production

deployment must be altered to use HTTPS. Your test code needs to make the

appropriatechangestotheURLbeingtested.

SecuringcommunicationstoMongoDB

Forperformancereasons,youwillwanttoplaceyourdatabaseinthesame

AWS datacenter as your Elastic Beanstalk Node app. As an added bonus of

doingthis,theabilitytosecurethecommunicationbetweenyourNode.jsservice

and your database becomes easier. This is because everything is sent over the

internal data center network and never gets out over the public Internet. As a

furthermeasureofsecurity,youcanalsocommunicateoveranSSLconnection

ifyouuseadedicatedplanfromMongoDBInc.

With a dedicated plan, you also define custom firewall rules so that your

databaseaccessislimitedtospecificIPaddressrangesand/ortospecificAWS

EC2securitygroups.

DNSandcertificatesetup

IntheIntroduction,yousawaphysicaltopologydiagram(seefigure4)that

showedadomainnamebeingroutedthroughaDNSserverthathadacertificate

toenableHTTPS.Youcannowgoaboutsettingthatup.

First, go to the Route 53 service management console. Click Register

Domain to start the process of getting your own domain, or you may need to

clickGetstartednowundertheheadingtodoadomainregistration,thenclick

RegisterDomain.Ifyoualreadyhaveyourowndomainnameandwanttouse

it, you can do that but that is not covered in this book. Here is the Route 53

managementconsole:

Figure62-Route53servicemanagementconsole

Youcantypeindifferentnamestotryandyouwillbeabletofindonethatis

available.There is an initial chargeas well as a small recurring fee when you

purchaseadomainname.ForthisimplementationoftheNewsWatchersample

app, I settled on newswatcher2rweb.com since newswatcher.com was already

taken.

Figure63-Route53domainavailability

It can take an hour or longer before everything is ready with your new

domainname.Onceitisready,youcangetacertificateallsetupthatusesthe

AWSCertificateManagerService.NavigatetoCertificateManagerandclick

GetStarted.Enterthedomainnamevariationsyouwanttobesupported.You

canusewildcardsandhavetheflexibilityyouneed.Hereisthepagewherethat

happens:

Figure64-AdddomainnamestoAWSCertificateManager

Afterthisscreen,thereareafewmoretoclickthrough.Atsomepoint,you

will also need to validate this action through an email you will receive that

verifiesthatyouaretheownerofthedomainnamethatthecertificateisbeing

setupfor.

AcertificatecannowbesetontheElasticBeanstalkloadbalancer.Youdo

thisthroughtheAWSmanagementportal.Herearethesteps:

1. Open your AWS Elastic Beanstalk environment and on the left,

findandselectConfiguration.

2. Click the gear by Scaling and change Environment type to be

Loadbalancing,autoscaling.

3. Wait for this change to take effectand then click Configuration

againfromtheselectionsontheleftandthenclickLoadBalancing.

4. SelectyourcertificatefromtheSSLcertificateIDdropdown.

5. ClickApplyatthebottomofthepage.

Nowyoucanwatchthestatusthereuntilitindicatesthattheconfiguration

changeissuccessful.

Figure65-ElasticBeanstalkstatus

NowyoucansetuptheDNSroutingtoyourElasticBeanstalkloadbalancer.

Go back into the Route 53 management console and click under the DNS

managementtheHostedzones link. Then click on your Domain name. Click

CreateRecordSet.

Figure66-AWSCertificateManagementCreateRecordSet

Fillouttheformontherightasshowninfigure69.ForAliasTarget,select

yourloadbalancer.Youcanaddarecordsetwithname“www”andthenmake

onewithablankname.

Figure67-Route53createrecordset

Youarealmostdone.YouneedtoturnoffHTTPaccessattheloadbalancer.

Todothat:

1. Go to the EC2 service management console and click 1 Load

Balancers.

2. ClicktheListenerstab,thenclickEdit.

3. DeletetheHTTPentry,thenclickSave.

Figure68-EC2serviceListenerstab

BackattheElasticBeanstalkenvironmentpage,ifyouclickontheURLat

thetop, itwill nolonger work.In yourbrowser, changethe URL tostart with

https:andyouwillbetoldthecertificatedoesnotmatch.NowtypeintheURL

tothedomainnameyouregisteredandyouwillseeeverythingupandworking.

HTTPSisnowworking.Ifyoutry,https://www.newswatcher2rweb.com/itwill

work,buthttp://www.newswatcher2rweb.com/willnotwork.

12.16Deployment

Atthispoint,youhaveeverythinginplacetostarttryingoutyourmiddle-tier

webserviceAPIthatisimplementedasanHTTP/Restendpoint.Youobviously

would not build a service like this without testing it along the way. All of the

discussioninvolving thetesting of theservice is discussedin the nextchapter.

You can see it deployed at this point and then understand the testing that is

necessary.

Don’tgetthewrongimpression.Icertainlywrotethecodeinsmalliterations

andtestedeach andeverybit ofitalong theway.Good developersiterateand

testeverythingastheygo.

At this point, you can zip up your code and deploy it up to AWS. Before

doingso,youcantestthingsoutwithsometeststhatwillbedescribedinchapter

13. You will actually want to test on your local machine as well as deploy to

someknownstagingcloudlocationandtestthereaswell.Youwouldberunning

thetestcodefromyourlocalmachinetogoagainstastagingsitethatishostedin

AWS.

HerearethefoldersandfilesIselectedtohavezippeduponmyWindows

machine.Thereisaright-clickselectiontocreateazipfileunderSendtoona

windowsmachine.

The build and public folders are not actually there yet, so leave those out.

TheyarelaterfoldersthatgetaddedthatholdtheReactapplication.

Youfollowthesameinstructionsdetailedpreviouslytogetyourapplication

code up and running. And now at this point have secured the traffic through

HTTPS.

Figure69-Selectionstomakeazipfile

Note:Ifyoufinderrorsupondeployment,youcanclickonthelogsselection

intheUIoftheElasticBeanstalkAWSconsole.Fromtheactionsmenu,selectto

downloadthefulllogs.Inthefolderyoudownloadyouwillfindafilenamedeb-

activity.log.Scrollandseeifthereareanyindicationsastowhythenpminstall

commandfailed.ThereseemstobeanongoingissuewithAWSElasticBeanstalk

installswheretheyrunoutofmemorywhentheinstallhappens.Toworkaround

this,switchtoat2.smallmachineandthentryyourinstallagain.

Chapter 13: Testing the NewsWatcher Rest

API

NowcomestheexcitingpartwhereyouwillgettoseetheHTTP/RestWeb

APIexercised.Youwillprovethattheserviceisupandrunninglocally,thentest

the deployment and verify everything in production. Once sufficiently proven,

youcanstartthefinaltaskofcreatingaUIforNewsWatcherandbeconfident

thattheintegrationwillgosmoothly.

Thischapter will present several practices thatyou will want to follow for

exercisingyourcodetofullytestit.Itisobviouslyalotsimplertotestanddebug

issueslocally.Let’sfirstlookathowtoemploytechniquesfordebuggingissues

inproduction.

13.1DebuggingDuringTesting

Let’sfirsttalkaboutdebuggingtechniques.Youwillneedtodosomeofthat

asyouruntestsandneedtoexaminetheexecutionofyourcode.

In some cases, output logging to the Node console will provide you with

enough clues to track down an issue. This means that you must log important

thingsthat are happening in theapplication. Beyond this, you willneed a few

toolstohelpyoudoyourinvestigations.

One tool at your disposal is the VS Code debugger. Before deploying

anything,you can run yourcode locally. Youcan useVS Code to debug your

Node.jsprojectcode.

IfyouwanttodebugyourNode.jscode,youopenyourproject,andlaunch

theNode.jsprojectbypressingF5.Youcansetupyourbreakpointsinadvance,

oraddonesasneededthatyouwanttobehit.Onceyourprojectisrunning,you

run Mocha from the command line and exercise your code through tests you

havewritten.Thenyoucanstepthroughyourcode.

Note:Therearesomeissueswhenforkingasecondprocesswhenrunningin

debug mode. That is why I have the line you can uncomment that makes it

possible. Uncomment “var node2 = cp.fork('./worker/app_FORK.js', [], {

execArgv:['--inspect=9229']});”.

TosetupdebugginginVScode,clickthedebugicon.Youwillseeagear

iconat thetop ofthe windowthat youcan clickto createthe launch.jsonfile.

ThisisthefilethatinstructsVSCodehowtoproceed.Youcansetituptohave

twoconfigurationsinthefile.Onewillbeforlaunchingyournodeprocesswith

debuggingcapability.TheotherentryisforattachingtoanalreadyrunningNode

process.Whenyouclickthegearicon,selecttheNodeselectionandcreatethe

file.Yourfilemightlookasfollows:

{

"version":"0.2.0",

"configurations":[

{

"type":"node",

"request":"attach",

"name":"AttachtoRemote",

"address":"TCP/IPaddressofprocesstobedebugged",

"port":9229,

"localRoot":"${workspaceFolder}",

"remoteRoot":"Absolutepathtotheremotedirectorycontainingtheprogram"

{

"type":"node",

"request":"launch",

"name":"Launch",

"program":"${workspaceFolder}/server.js"

{

"type":"node",

"request":"attach",

"name":"Attach",

"port":9229

}

]

}

Totheleftofthegearisthestartbutton(youcanalsopressF5).Thedrop-

down menu will show you the config section options that come from your

launch.jsonfile. The far right greater-thansymbolin the box opens the output

console window. It is a good idea to always have that open to view any

statementsorerrorsthatgetdisplayed.

Figure70-VisualStudioCodedebuggin

Youcanopenthelaunch.jsonfileandlookatit,butyoudonotneedtomake

anyadjustments to it as the defaultsare just what you need, unlessit is set to

startnodewithanotherfile.Youcanlookupthedocumentationonthesettings

that can be used in the launch.json file on the Node website and VS Code

Website.

Toplaceabreakpoint,openaJavaScriptfileandclickouttotheleftofthe

margin, or click on the line and press F9. Once you hit a breakpoint while

runningcode,yougetfullaccesstoinspectthecallstackandvariables.

13.2ToolstoMakeanHTTP/RestCall

YouwillprobablywanttouseatooltomakeindividualcallstoyourAPI.

Thatway,youcantakesmallstepstogeteverythingverifiedbeforeyouthrowa

testharnessintothemix.IhaveinstalledPostman,FiddlerandCurl.exeonmy

machine for testing individual HTTP/Rest calls. There are many of these such

tools.Withthesetools,youcansendHTTPrequeststoyourserviceandviewthe

returnedresponse.

Fromoneoftherecommendedtools,youcansetuptheverbusage,aswell

astheheadersandtheJSONbodycontent.Youcandoasendandthenlookat

thereturnedresponse.Let’slookattheUIofFiddlerandIwillexplainthebasics

ofhowtouseittocallyourAPI.

YoucansendanHTTPrequesttotheroutehandlerthatregistersauser.This

is an obvious first place to start. To do that, you know you have the

app.use('/api/users',users)callinyourserver.jsfilethattakesyoutotheusers.js

function of router.post('/', function(req, res, next) {}. This is the code that

createsanewregistration.

YouwillwanttomakeapostverbcallandpassinaJSONbodythatcontains

thedisplayname,emailandpassword.Youthenexpecta201returncodetobe

givenback to indicate a successfulcreation. In the return responsewillbe the

returneddocumentthatyoucouldinspect.

YoucannowstartupyournodeservicewithyourprojectopenandpressF5

todebug,orpressCtrl+F5torunwithoutdebugging.Thefirstthingyounoticeis

that you get a console app window that opens in the VS Code UI. That

represents your node process running and the executing of your server.js file.

Youwillseeallofyourconsoleloggingappearinthiswindow.Ifyoudon’tsee

it,clickonthetoolbaronthelefttogettothedebugareaandyoucanopenit

fromthere.

YourNode.js server is running as a local process now. To make an HTTP

requestyoucanconnecttolocalhostwiththeportnumberof3000andinteract

withtheRestAPI.YoucanopenFiddlerandtrythisout.Thefirstthingtodois

tolimitwhattrafficFiddlerseesgoingbackandforth.Otherwise,youwillget

lostinthestreamoftraffic.Todothisandalso makeanHTTPrequestdothe

following:

1. IntheFiddleWebDebugger,clicktheFilterstab.

2. Inthetopsection,checkUseFilters.

3. In the Request Headers section, check Show only if URL

containsandtypein“localhost”.

Figure71-Fiddlerfiltering

4. Clickthedrop-downmenuX->Removeall to get rid of all old

traffic.

Figure72-Fiddlerremoveall

5. ClicktheComposertab,thenclicktheOptionssub-tabunderthat

andmakesureitlooksasfollow:

Figure73-Options

YouwantFiddlertofigureoutthecontentheaderlengthvalueforyou.

6. ClicktheRawtab,andenterthefollowingrequest:

POSThttp://localhost:3000/api/usersHTTP/1.1

User-Agent:Fiddler

Host:localhost:3000

Content-Type:application/json

Content-Length:85

{

"email":"bush@sample.com",

"displayName":"Bushman",

"password":"abc1234#"

}

Figure74-FiddlerRAWrequest

7. ClickExecute.

Ifyoulookintheleft-handpane,youwillseetherequestbeingsentanda

responsereturned.Ifyoudouble-clickthe201response,youcanexamineitand

seethatitworked.ItshouldlookasfollowsifyouclickonRaworJSONview:

Figure75-Fiddlerrequestandresponse

You have now successfully seen your API exercised. Open the MongoDB

Compass tool and you can also see that there is a new document created. In

Compass,youwillseesomethinglikethefollowing:

Figure76-Atlasmanagementportalwithanewdocument

Youcan now enter every single request through Fiddler to prove them all.

Next,trytologinandgetatokenback:

Figure77-Fiddlertologinandgetatokenback

Lookattheresultandyouwillseeatokenintheresponseifyouopenitup.

Youwillneedthistokentouseinthenextrequests.Youcannextmakeacallto

retrieve the user document and see if there are any news stories that have

matchedthefilter.

Figure78-Fiddlertoseethetoken

InFiddler,ClickExecuteandyouwillseearesponsewiththereturneduser

documentandpossibly somenews stories.Thisis greatprogress, andyou can

continuetotryallofyourAPIcallsanddebugeachoneifneeded.Asyouget

eachoneworking,youcancreateatestcaseforeach.

As was mentioned, you might find a tool other than Fiddler that you like.

Curlisa niceoptionbecause youjustopen acommandprompt andexecute it

there.Ithasasimplesyntaxthatyouusetoformulateyourrequests.

13.3AFunctionalTestSuitwithMocha

NowyouwilltakethenextstepofautomatingthetestingoftheAPIusinga

test suite. This will then become your functional test pass. You can start by

implementing some of the same operations that were already tried when you

used Fiddler. To start with, you need to add a new folder to your VS Code

projecttoholdthetests.Youcanalsosetuptheaddednodemodulesyouneed

forthetests.

Note: NPM lets you download different types of modules. Like previous

modulesI’veshownyou,youdownloadthemandthenusearequirestatementto

utilizethemascodemodules.Others,likeMocha,arenotcodemodules,butare

command-line tools. What you do is write a JavaScript file that Mocha will

interpretandrun.Youthenuseacommand-linetoexecuteMocha,anditdoesits

work. Mocha is installed as a local part of your project. This means the

executable will be referenced from that location. Mocha can also be installed

globallyifyoulike.

Addthefollowingtothepackage.jsonfile:

"devDependencies":{

"mocha":"^3.4.2",

"supertest":"^3.0.0"

}

The devDependencies section is reserved for non-production modules that

youwillnotneedtodeploytoaproductionbuild.Theyareonlyneededtorun

yourtestcodelocally.

We cannow go throughsome of thefiles in thetest folder tosee how the

NewsWatcherapplicationcanbetested.ThisfirstfilewillbeusedtohittheAPI

endpointtoexercisetheroutes.

Writingmochatests(functional_api_crud.js)

You will make use of the supertest module and the assert module. The

necessaryrequire()callscanbesetupforthose.Theserverobjectalsoneedsto

be retrieved and given time to be initialized. There are special blocks of code

thatmochawillrunbeforeandafteralltests.Therestofthecodemakesuseof

mochatestblockstoruntestswith.

Therequestobjectiswhatyousetupfromthesupertestmodule.Withthat,

youmaketheHTTPpostverbcallandregisterauser.

You will use the describe keyword for any major test block and use the

individualitkeywords for each test inside of that. With just one test to run, it

wouldlookasfollows:

varassert=require('assert');

varapp=require('../server.js');

varrequest=require('supertest')(app);

describe('Usercycleoperations',function(){

//Waituntilthedatabaseisupandconnectedto.

before(function(done){

setTimeout(function(){

done();

},5000);

});

//Shuteverythingdowngracefully

after(function(done){

app.db.client.close();

app.node2.kill();

app.close(done);

});

it("shouldcreateanewregisteredUser",function(done){

request.post("/api/users")

.send({

email:'bush@sample.com',

displayName:'Bushman',

password:'abc123*'

})

.end(function(err,res){

assert.equal(res.status,201);

assert.equal(res.body.displayName,"Bushman","Nameofusershouldbeasset");

done();

});

Note:Mochaactuallysupportsseveraldifferentstylesofsyntax,sodon’tbe

confusedifyouseeotherprojectsusingMochaanditdoesnotlookexactlylike

thecodeinthisbook.

Iwillgothroughthepreviouscodetomakesureyouunderstandit.Tostart

with,youhavetheusualNode.jsrequirestatementsatthetop.Forthesupertest

module,youspecifythatyouaregoingtohitthepassedinwebserverendpoint

ofyourNodeproject.Thestringsyousetasparameterstodescribe()andit()are

printed out for you as part of the test run. You should use text that helps you

rememberwhatyouaretesting.

Withthiscode,youhaveatestthatwillverifythatyoucanregisterauser.

Theitblocktakesastringtodescribewhatyouaretestingandthenafunctionto

run.Thereisadone()functionthatyoucalltosignaltomochathatitcanmove

onto the nextit block. Thesetests are each runsequentially. If youdon’tcall

done(),thetestwilleventuallytimeout.

Tousesupertest,youspecifyaverboperationtouse.Thisoneisusingpost.

Youstringtogetherthefunctioncallssend()andend().Eachonewillgetcalled

in sequence. The sendfunction does the HTTP/Rest request and has the body

set.

The end() function can get the response and validate the return code and

valuesfromthereturnedbody.Youusetheassertmoduleforvalidations.

Therearesomecaseswhereyouhaveasecondtestthatreliesontheresults

ofthefirsttest.Thiscanbetrickyifthereisdelayedprocessingofthefirsttest

code. You can either stack one test inside the other, or use a JavaScript

setTimeout()calltodelayyoursecondtestrunbyabitandthenhaveitrun.

Forexample,whenauserchangesanewsfilters,thatoperationwillreturn

immediately. This means that the test code will move on to the next test. The

way NewsWatcher works, is that it sends a message to the forked process to

updatethenewsstories.Ifyouwantedtotestachangetoaprofilefilterstring,

youwouldrealizethatitwouldtakeasecondtodothatinthebackgroundand

youwouldputinadelaybeforethenextcodecouldran.

Thefollowingcodeshowsusinga delayofthreesecondsbefore runninga

test:

it("shouldallowaccessifloggedin",function(done){

setTimeout(function(){

request.get("/api/users/"+userId)

.set('x-auth',token)

.end(function(err,res){

assert.equal(res.status,200);

savedDoc=res.body.newsFilters[0].newsStories[0];

console.log(JSON.stringify(savedDoc,null,4));

done();

});

},3000);

});

Hereisamorecompletesetofteststhatregistersauserandthenmakessure

theycanloginandthendeletestheaccounttocleanthingsup.Thetestrightat

thestartistoverifythatapersoncannotloginiftheydon’tfirstregister.

There may be negative tests you can put in place to verify your error

handling code. In this code below, I make use of local valuables inside the

describe block to pass these between tests such as the token variable. For

example, you need to capture the token at sign in time and keep using it on

subsequentrestcallsthatyouaretesting.

varassert=require('assert');

varapp=require('../server.js');

varrequest=require('supertest')(app);

describe('Usercycleoperations',function(){

vartoken;

varuserId;

varsavedDoc;

//Waituntilthedatabaseisupandconnectedto.

before(function(done){

setTimeout(function(){

done();

},3000);

});

//Shuteverythingdowngracefully

after(function(done){

app.db.client.close();

app.node2.kill();

app.close(done);

});

it("shoulddenyunregistereduseraloginattempt",function(done){

request.post("/api/sessions").send({

email:'bush@sample.com',

password:'abc123*'

})

.end(function(err,res){

assert.equal(res.status,500);

done();

});

it("shouldcreateanewregisteredUser",function(done){

request.post("/api/users")

.send({

email:'bush@sample.com',

displayName:'Bushman',

password:'abc123*'

})

.end(function(err,res){

assert.equal(res.status,201);

assert.equal(res.body.displayName,"Bushman","Nameofusershouldbeasset");

done();

});

it("shouldnotcreateaUsertwice",function(done){

request.post("/api/users")

.send({

email:'bush@sample.com',

displayName:'Bushman',

password:'abc123*'

})

.end(function(err,res){

assert.equal(res.status,500);

assert.equal(res.body.message, "Error: Email account already registered", "Error should be already

registered");

done();

});

it("shoulddetectincorrectpassword",function(done){

request.post("/api/sessions")

.send({

email:'bush@sample.com',

password:'wrong1*'

})

.end(function(err,res){

assert.equal(res.status,500);

assert.equal(res.body.message,"Error:Wrongpassword","Errorshouldbealreadyregistered");

done();

});});

it("shouldallowregisteredusertologin",function(done){

request.post("/api/sessions")

.send({

email:'bush@sample.com',

password:'abc123*'

})

.end(function(err,res){

//<Session&Cookiecode>cookies=res.headers['set-cookie'];

token=res.body.token;

userId=res.body.userId;

assert.equal(res.status,201);

assert.equal(res.body.msg,"Authorized","MessageshouldbeAUthorized");

done();

});

it("shouldallowregisteredusertologout",function(done){

request.del("/api/sessions/"+userId)

.set('x-auth',token)

.end(function(err,res){

assert.equal(res.status,200);

done();

});

it("shouldnotallowaccessifnotloggedin",function(done){

request.get("/api/users/"+userId)

.end(function(err,res){

assert.equal(res.status,500);

done();

});

it("shouldallowregisteredusertologin",function(done){

request.post("/api/sessions")

.send({

email:'bush@sample.com',

password:'abc123*'

})

.end(function(err,res){

token=res.body.token;

userId=res.body.userId;

assert.equal(res.status,201);

assert.equal(res.body.msg,"Authorized","AuthorizedMessage");

done();

});

it("shoulddeletearegisteredUser",function(done){

request.del("/api/users/"+userId)

.set('x-auth',token)

.end(function(err,res){

assert.equal(res.status,200);

done();

});

You run your Mocha test suite from a command prompt. On Windows

machines you open a Node.js command prompt. You can also use GitHub

DesktopandrightclickyourrepositoryandselectOpeninGitshell.Onceyou

areatthecommandprompt,navigatetothelocationofyourprojectifneeded.

ThisisnotnecessaryifyouopenupapromptfromGitHubdesktop.

Youdon’tneedtostartupthenode.jsapplication,becausetheMochacode

will do that. You can run Mocha from the local project folder in the git shell

windowasfollows:

./node_modules/.bin/mocha--timeout30000test/functional_api_crud.js

Youmayneedtoplayaroundwiththetimeoutargument.Itispossibletoget

falsefailuresbecauseofMochatiming-outand movingontothenext itblock

tooquickly.Theoutputforthecompletefunctionaltestsuitelooksasfollows:

Figure79-Mochafunctionaltestoutput

Note: You should realize that, even if you are running against your local

Node.js service, you are still hitting the real AWS hosted MongoDB database.

YoumightneedtoopentheCompassappanddeleteunwanteddocumentsthat

youcreatedthroughyourtests.ThetestsincludedwiththeNewsWatchersample

are written to clean up after themselves. Of course, you could install a local

copyofMongoDBonyourmachineandconnectagainstthataswell.

TorunagainstthedeployedNode.jsapplicationinAWS,youcanchangethe

supertest usage to go against the production URL, such as

https://www.newswatcher2rweb.com. You can see in the GitHub project that I

haveafewcommentedoutlinesfordifferentwaystorunthetests.

13.4PerformanceandLoadTesting

Writinganapplicationthatcanserveasingleuserisnotabigchallenge.The

realchallengecomeswhenmultiplepeopleareallhittingthewebserviceREST

APIatthesametime.

To write the NewsWatcher sampleapp and get it towork for a singleuser

wasjusttwoweeksofworkforme.Togetittoscaleandhandlethesimulated

loadof many users, requiredmuch longer than thatto work out allthe issues.

You certainly don’t want to wait until your big production rollout to find that

yourcodefallsflatonitsfacewhenmorethanonepersonusesit.

Howareyougoingtoaccomplishtestingyourcodeatscale?Theonlyway

to accomplish this is with a test suite that can provide usage in parallel and

simulatemultipleusers.

ThereareUItestingtoolsthatcanrecordyourusageofawebsiteandreplay

it. They can be replayed more than once at the same time to simulate lots of

interaction happening. This might work well for some sites that serve static

contentand have no concept of people logging in and exercisingsome unique

workflowinthebackendservicelayer.

Loadtestingenablesyoutoprovethescalingofyourapplication.Youalso

use this to measure your SLA (Service Level Agreement) values under a

constant load. You can experiment by increasing the load until you find the

breakingpoint.Thiswilltellyoutheabsolutepeakvaluesyoucanrununder.To

dothis,youneedtoaltertheURLtobethatoftheproductionorcloudstaging

environment.

Running the load testing suite can also help you test out your Elastic

Beanstalkscalingstrategiesandtopology.Itwillofcoursebeusefulinverifying

theperformanceofMongoDBthatyouhaveworkedhardtooptimize.

13.5RunningLint

You can install the npm packages "eslint" and "eslint-plugin-react". These

allow you to run validations that scan the code and look for syntax errors,

uninitializedvariablesandevenspecificstyingthatyouwant.Iwouldsaythatit

isarequirementtohavethisinplace.Ifyouwerewritinginalanguagesuchas

C++thatwascompiled,yougetthatalldoneupfront.Unfortunately,youwrite

codewithJavaScript,butthendon’tgettoseeerrorsuntileachlineexecutes.

Youwillseethatthepackage.jsonfilehassomescriptssetuptorunlintand

alsocombinetherunningoflintwiththerunningofthefunctionalmochatests.

Thisiswhatisinthepackage.jsonfileforthat:

"scripts":{

"lint":"eslint**/*.js",

"test":"mocha--timeout30000test/functional_api_crud.js",

"pretest":"npmrunlint",

"posttest":"echoAlltestshavebeenrun!"

}

Whenyoufirstinstalleslint,youcanrunitininitializationmodeanditwill

createafilebasedonthequestionsyouanswer.Thecommandis:

./node_modules/.bin/eslint–init

Iranthisandthenmadeafewtweakstothefollowingtwofiles:

//.eslintignore

node_modules

build

src

test

Andtheotherfile:

//.eslintrc.json

{

"env":{

"browser":true,

"es6":true,

"node":true

"extends":"eslint:recommended",

"parserOptions":{

"ecmaFeatures":{

"experimentalObjectRestSpread":true,

"jsx":true

"sourceType":"module"

"plugins":[

"react"

"rules":{

"linebreak-style":[

"error",

"windows"

"no-console":"off"

}

It is as simple as running the following commands. The first just runs lint

andthesecondrunslintandthenthetests.Lintwilllistalltheerrorsandyoucan

keepfixingthemuntiltheyareallgone.Herearethecommands:

npmrunlint

npmtest

Chapter14:DevOpsServiceLayerTips

It is a fabulous accomplishment to get the code all tested and deployed to

production. Don’t get too comfortable though, as it is quite another matter to

managetheoperationsofafull-stackapplication.Thischapterwillpresentsome

key skills that will make your life easier when it comes to running a 24x7

operationsforyourservicelayer.

Chancesarethatyouwillexperiencesometypeofcatastrophicfailurebefore

too long. First off, you absolutely want to do everything up front to put

preventativemeasuresinplace.Astheoldsayinggoes“Anounceofprevention

isworthapoundofcure.”

Youalsoneedtoput aplaninplaceforhandlinga crisiswhenithappens.

Youwanttobeinapositiontohavealloftheinformationatyourfingertipsto

make it possible to recover in the shortest amount of time. There are some

generaltechniquesthatwillbepresentedhere,butyouwillhavetocomeupwith

yourownspecificstrategiesthatfityourownenvironment.

Letmemakeabriefcommentaboutcontinuousintegrationandcontinuous

delivery(CI/CD).Ifyouareworkingonanykindofsubstantialprojectthatwill

begoingonforawhile,orhasmultiplepeoplecontributing,youdefinitelyneed

to implement CI/CD. Manually performing the tasks of building, testing, and

deployingcodegetsoldfast.Alwaysrememberthatdoingthesethingsmanually

is prone to human error. If your DevOps process is not automated with full

integrationtesting,thenitisnotreallycomplete.

Thereisawisesayingthatstates“Youhavetoslowdowntospeedup.”You

caninterpretthattomeanthatalittleinvestmentupfrontpayshugedividends

over and over. This ability to centrally coordinate CI/CD really helps groups

withanagileprocessiteratemorerapidly.Productivitygoesupbecauseofthis

automation and team downtime is reduced because integration bugs are not

spreadacrosstherestoftheteam.Itismuchmoreexpensiveintimeandmoney

tocatchbugsinproduction,sothereisahugesavingshere.

14.1ConsoleLogging

Writingmessagestoaconsoleoutputor logfileisanage-old practicethat

mightbeusefulifyoucanavoidbeingoverwhelmedbytoomuchloggingand

thenbeabletointerprettheinformationtosolveproblems.

NodehastheConsolemodulethatisusefultowritetraceoutputto.Thisis

available in your application already, so you don’t need to use a require

statement. Here are some useful methods you can use that are found on the

consoleobject:

log()

info()

error()

warn()

Basicmethodforoutputtingtext.Itcomesinseveraldifferent

formsallwiththesamefunctionsignature.

console.log(“wemadeithere%d”,someVarNumber);

dir()

Itisusefultoviewanobjectyoumighthaveinyourcode.There

areoptionsavailableforthis,forexampletorecursefurtherthanthe

defaultdepthof2levels.

console.dir(someObject);

time()

and

timeEnd()

Tologelapsedtime,youusethesetwomethods.Youwillgetthe

elapsedtimewhenyoudothefollowing:

console.time(“start”);

//somecodeoperationsthatyouwanttotime…

console.timeEnd(“start”);

trace()

Forshowingastacktracefromthepointinyourcodewherethis

iscalled.

console.trace(“someLabel”);

assert()

Thisisthestandardassertionusagecommonlyavailable.

console.assert(valid,“Hi”);

You can set up logging to go into a file that can be looked at. Sometimes

loggingwillpointyouinthegeneraldirectionandthenyoucanusethedebugger

to further diagnose an issue. As another option you could send every log

message to a special collection in MongoDB. You can set up MongoDB

documentstohaveatimetolive(TTL)beforetheyareautomaticallydeletedor

setthecollectionascapped.

14.2CPUProfiling

TheV8enginecanprovideyouwithCPUusagereports.Thesimplestwayto

dothisistolaunchyournodeprocesswiththeprofileflagset,afterrunningyour

testcode,youstoptheprocessandyouwillthenhaveafileavailabletoview.

YouwouldlaunchNodeasfollowsonyourlocalmachine:

node--profserver.js

Ifyouthenrunsometestcode,suchasyourloadtestingsuite,youcanget

some idea of where your code might be spending most of its time. Once you

have run your test code for a bit, you stop the node process and a file named

somethinglike“isolate-000001D213C4D490-v8.log”willbesaved.

Next, open your browser and navigate to the Chrome V8 profiling log

processor site at http://v8.googlecode.com/svn/trunk/tools/tick-processor.html.

Youcanopenyourlogfilewiththistoolandseeanice,crisplayoutofyourcode

callslistedinorderofwheremostofthetimewasspent.Thereisacommand

lineversionofthetickprocessoralsoavailablefordownload.

Youwillseereportsthatlookasfollows:

Figure80-V8Reportsample

Figure81-V8Reportsample

You can also utilize the V8-profiler using a Node module to start it up in

yourcodeifyousetuproutehandlersandwanttoprofilethingsinproduction.

TheV8engine,whichisoneofthecomponentsthatNode.jsisbuilton,letsyou

run an analysis of production running code. Profiling data can be sent to an

externalfilethatyoucanlateropenandviewtheresultswith.Hereisthecodeto

accomplishaprofilerun:

varfs=require('fs');

varprofiler=require('v8-profiler');

profiler.startProfiling();

...dosomeprocessing...

varprofileResult=profiler.stopProfiling();

profileResult.export()

.pipe(fs.createWriteStream('profile.json'))

.on('finish',function(){

profileResult.delete();

});

Here is a screenshot of the Chrome Dev Tool (press F12 while running

Chrome)thatiscapableofreadingtheprofilefile.

Figure82-ChromedevelopertoolV8profilereport

Toreadinyourfile,ClicktheProfilestab,thenclicktheProfilespaneand

clickLoadtoloadyourfile(s).

Withprofilingturnedonwhilealoadtestwasrun,Iwasabletoproducea

reportthatshowedafewissuesIcouldaddress.Oneofthemturnedouttobethe

useofbcryptfor password hashing. As seen in the profile analysis, you see a

bottom-upviewofthecallingtree.Ifyouexpanduntilyouseeyourfunctions,

youcanthenunderstandwherethecallisbeingmadefrom.Here,youseethat

thebcrypt.hashSync()functioncalltook19%ofthetime.

Figure83-Profiledrill-down

Hereisthecodeforwhatyouseelistedforthefirstissue:

findUserByEmail(req.db,req.body.email,functionfindUser(err,doc){

...

passwordHash:bcrypt.hashSync(req.body.password,10),

Itwasobvious,giventhenameofthefunction,thatIwasnotusingtheasync

version of the hash-generation function. Go figure, I should have known that

hashSync() was not a good idea to call. As you know, you want to minimize

CPUusageonthemainNodethread.Checkingthedocumentation,Ifoundthe

asyncversionandmadethechangetouseit:

bcrypt.hash('bacon',8,function(err,hash){

});

Note:Itishelpfultogivenamestoallanonymousfunctions,otherwiseyou

willjust see a lot of “anonymous” functions and it is harder to pinpoint what

functionsareintheprofilelisting.

14.3MemoryLeakDetection

Inamanagedlanguageframework,youdon’tdirectlyallocatememoryand

subsequentlyfreeitup.Youcandoanewanddeleteofanobject,butyoustill

don’thavecontroloverthatmemory,suchastheactualreclaimingofit,ordoing

thingslikehavingmemorypointersintoit.Instead,thereisagarbagecollector

thatkeepstrackofmemoryreferencesandtheGCdecideswhentorunandwhen

memorycanberecycled.

The truth is, that with garbage collection running, you can still run into

memoryleaksthatwilleventuallycauseyourapplicationtoeitherrunslowly,to

completelyfreeze,orcrash.

Youcanobviouslycreateamemorygrowthproblemifyouhadsomethingas

simpleasanarraythatyoucontinuallypusheddataintoandneverfreeup.Ifyou

held on to an object reference permanently after you no longer needed it, the

garbagecollectorwillnoteverreclaimit.Athoroughcodereviewofcallbacks,

closures, constructor functions, and arrays can be a starting point to finding

memoryleaks.

Ultimately,yourbrainmightnotbeabletotracethroughalloftheintricacies

ofyourcodeandyouwillneedtotakememorysnapshotsthatcanbecompared

acrosstime.Node.jsapplicationsarealwaysbuiltwithseveral,ifnotdozens,of

downloadedmodules.Youhavetobesuspiciousofthose,aswell,astheymight

containmemoryleaks.

Youcan watchthe OS reporting of memory for your Node.js process over

time and see what kind of graph you have and, if you see an ever-increasing

amount of memory being taken up, you can then dive in and investigate. It is

even not unheard of for people to resort to restarting their Node.js processes

everydayjusttocircumventanymemoryleakproblems.Youmight,inreality,

nothavealeakif,overtime,youcanseethegarbagecollectorkickinanddoits

job.Comparememorysnapshotsovera24-hourtimespan.

You can use the v8-profiler module that was previously used for CPU

profiling to take memory snapshots. You can likewise have the output files

viewedintheChromedebugger.

varfs=require('fs');

varprofiler=require('v8-profiler');

varsnapshot=profiler.takeSnapshot();

snapshot.export()

.pipe(fs.createWriteStream('snapshot.json'))

.on('finish',snapshot.delete);

Note:Rememberto neverrunyour CPUprofiling atthe sametimeas you

take heap snapshots. The overhead memory usage for the CPU profiler will

inundateyou.

I have set up a specific route through Express that can trigger a memory

snapshot.ThatwayIcaninsertthisabilityintomytestsandhaveitavailablefor

measpartofaloadtestingrun.Hereisascreenshotofamemorysnapshotfile

thatisloadedintotheChromebrowserdebugger.

Figure84-Chromedevelopertoolwithmemorysnapshot

The Distance column shows you how many steps removed from the root

objectthememoryreferenceis.Youcanusuallyassumethattheobjectwiththe

shortest distance is the one causing a memory leak. The Shallow Size is just

whatthisoneusageistaking.RetainedSizegivesyouallofthespacethatwould

befreed up thatthis object isreferencing and thus holdingon to. Thisis only

trueifthoseobjectsarealsonolongerreferencedbyanythingelse.

Therearedifferentviewsyoucantryout,suchasContainment,whichhelps

youalsoviewlow-levelmemoryinternals.

14.4CI/CD

Doyouworkonaprojectwherethetasksofbuilding,testinganddeploying

codearealldonemanually?Whyisthat?Frommyexperience,thereasonteams

don’tautomate manual tasks is typically because they either lack incentive, or

they lack the knowledge. Lacking incentive is really a poor excuse. There is

plenty of incentive if you honestly look at the return on investment of

implementingaCI/CDprocess.Ifyour DevOpsprocessisnot automatedwith

fullintegrationtesting,thenitisnotreallycomplete.

Ionceheardaspeakerataconferencesaytodevelopers–“Itain’tdoneuntil

itisautomated”.Hewassayingthatyouneedtoautomateeverythingfromthe

check-in,testingandintegrationthroughbuildinganddeploying.Inthisbook,I

willsaythe‘D’inCDstandfordelivery.Thismeansthatcodeisservedupand

readytobedeployed.Wewill,however,taketheprocessallthewaythroughto

deliveryandcansayweimplementedaCI/CD+solution.

AnotherstatementIhaveheardandthatIoftenrepeatisthatof–“Youhave

toslowdowntospeedup.”Youcaninterpretthattomeanthatalittleinvestment

upfrontpayshugedividendsoverandover.Thisabilitytocentrallycoordinate

CI/CDreallyhelpstheagileenvironmentiteratemorerapidly.Productivitygoes

up because of the automation and team downtime is eliminated because

integrationbugsarenotspreadtotherestoftheteam.Itismuchmoreexpensive

intimeandmoneytocatchbugsinproduction,sothereisahugesavingstobe

hadhere.

Nowthatyouhavetheincentive,let’sassumethatitreallycomesdownto

knowledge.Afterreadingthis,youwillbeinformedandwillnolongerhaveany

moreexcusespreventingyoufromimplementingCI/CD.

ManytoolsandwaystoaccomplishCI/CD

Implementing CI/CD can, of course, be a complex undertaking. No one

articleor bookcan tellyou everythingyou need toknow becausethere areso

manytoolsoutthere.Noonesolutionwillworkforallprojects.Allprojectsare

unique in their code structure and frameworks used. We will only scratch the

surface and only do so for a Node.js project in a very narrow niche. We also

narrowourfocusdownbyrelyingonaPaaSsolution.

IfyouchoosenottogotherouteofsettingupacompleteCI/CDtool,you

can still automate all of this on your own and launch things manually. For

example,youcanimplementaseriesofGulptasksthatwoulddoalotofwhata

CI/CDsystemwoulddo.

Note: You can do everything in your DevOps environment through the

commandline.Onceagain,IhavechosentotaketherouteofusingUIswhere

available.

Youmusthavedevelopertestedcode

As a preparatory step, a developer will have prepared some code and

privatelytestedanynewfeaturebeforesubmittingittotheCI/CDprocess.The

testing should be as thorough as possible in the context of the rest of the

integratedsystem.Ifthatisnotpossible,thenitcanbetestedasanisolatedunit,

perhapswithstubbedoutormockedfunctionalityinjected.Thetestingshouldbe

runnablebyadeveloperbytyping“npmtest”,whichwillrunlintandthenall

testsforthatpartofthecode.

Once the developer has had their own private verification completed, they

can do their code check-in that will then be staged to go into the pipeline for

consideration.IfyouareusingGitandGitHub,thecodewouldbepushedtothe

branch you designate for the CI/CD to be triggered from. You would not

necessarily push it into the master branch. The master branch should only be

usedforthecodethatisrunninginproduction.Youcouldpushtomasterandtag

thatbutnothaveitdeployedtoproduction,buttotestandthenhaveastepthat

requiresmanualacknowledgmenttopushthattoproduction.

CI/CD is all about increasing your iteration speed and the quality of

everythingwritten.Ofcourse,youhavetoprovidehigh-qualitycomprehensive

testsuitestoachievethis.Onceyourcodeisreadytocommit,thenthepipeline

workflow of CI/CD takes place. So what does the cycle look like? Let’s go

througheachofthesteps.Hereisasimplediagramtoshowyouthepiecesthat

wouldmakeupasimpleNode.jsCI/CDsystem:

Figure85-???

AWStools

TherearealotofcompaniesmakingtoolstohelpwithdoingCI/CD.Ihave

lovedusingtheopensourcetoolJenkinsinthepastanditisagoodchoice,and

thereareevencompanieshostingitinaPaaSenvironmentifyoudon’twantto

installandmanageityourself.

This book will use AWS CodePipeline as the overall orchestration tool to

mage the build, test and deployment steps. The build and test are done with

AWSCodeBuild.DeploymentcanbedonewithCodeDeploytodeploytoECS

foraMicroservicesarchitecture.Inourcase,weuseElasticBeanstalk.Thereisa

directintegrationtothatinCodePipeline,soCodeDeploywillnotbeused.

StepOne:Install

Once the code is committed into the source code repository then the code

needs to be staged in the overall application. CI/CD tools actually are able to

listen to GitHub and will kick off immediately upon seeing a code push. The

ideaisthataCI/CDsystemcanseethepushanddoalocalcloneofthebranch

soyouhave afreshlyintegrated codeversionto nowwork frominthe CI/CD

system.

AllcodedependenciessuchasNPMmoduleswouldneedtobeupdatedin

thiscloneandthatispartofwhattheCI/CDwouldkickoff.Thisisdonewithan

npminstallcommand.Sincewearewantingtogetaproductionenvironmentup

andrunning,wewillhavetodoanextranpminstalltogettheDevDependencies

sincethosedon’tgetinstallediftheenvironmentvariableNODE_ENVisequal

to“production”.Thiswillbeexplainedagainaswegothroughthesteps.

StepTwo:Build

Onceallthedependenciesareinstalled,wecandoanybuildandteststeps.

In our case, the Node.js code is set to go, but there would be a build step

required for our React code. Then things like Lint, can be run. If Lint passes,

thentheMochatestscanberun.

ComprehensivetestingiscentraltogettingCI/CDworking.Thisstepcould

runathoroughsuiteoftestsatalllevels,suchasUnit,integration,feature,load,

performance and UI automation testing. A perfect pass is expected and you

would get generated reports to show that all went well and also produce code

coveragereportingandhavememoryleakdetection.

StepThree:Post-Buildanddeploy

This is the final step that can generate the files necessary to deploy and

actuallydothedeploymentifyoulike.Sincewehaddevelopmentdependencies

likeLintandMochainstalled,wecanbackthoseoutandjusthavedependencies

neededfortheproductionenvironment.Thedeploymentcanbeautomatic,oran

email can be sent to a person to do manual acknowledgment to approve it. If

your team is in favor of TIP (Testing in Production) you could deploy to a

reservedhiddenportionofyourproductionenvironment.

You should have a testing environment that is as close to production as

possible.Onethingtorememberistoalsohaveinplaceadedicateddatabasefor

testing purposes hosted in PaaS that is always available. You do not need to

continuallydeploythereifitissimplythelocationofthedataandnothingneeds

tobepreconfigured.YoucanrunsomecleanupscriptaspartoftheCI/CDstep

here if that is necessary and also some pre-population script if you require

certainDocumentstobeinplace.PaaSofferingsforMongoDBandothersare

availableasdocument-baseddatabases.

Ifthetestingfails,youwouldbenotifiedandtheCI/CDprocesswouldnot

proceedanyfurther.Youwouldwantfailuresautomaticallyenteredintoanissue

trackingsystemtoofficiallytrackandresolveissues.

NewsWatcherCI/CDwithAWSCodePipeline

Here is a quick introduction to how to set up a CI/CD process with AWS

CodePipeline.

MakesureyouarealreadysignedintoGitHub,andthenintheAWSconsole

select CodePipeline from the Services drop down. Here is a brief listing of

someofthesteps.Ihaveleftoutsomeofthedetailsastheyshouldbeobvious.

1. ClickGetStarted.

2. NameyourpipelineandclickNextstep.

3. For "Source provider" select GitHub and click Connect to

GitHub.

4. ClickAuthorizeaws-codesuiteandconfirmyourpassword.

5. Select the Repository and Branch. I selected the master branch.

ClickNextstep.

6. SelectAWSCodeBuildfortheBuildprovider.

7. SelecttheruntimetobeNode.jsandthelatestversion.

8. FortheBuildspecification,selecttouseabuildspec.ymlfile.

HereiswhatIhaveinthebuildspec.ymlfile:

phases:

install:

commands:

-echoInstallingNodeModules...

-npminstall

-npminstall--only=dev

build:

commands:

-echoBuildstartedon`date`

-echoBuildingReactWebapplication

-npmrunbuild-react

-echoPerformingTest

-npmtest

post_build:

commands:

-echoFinalbuild,withoutdevDependencies

-rm-rfnode_modules/

-npminstall

artifacts:

files:

-'**/*'

Once you are ready to go, you can merge and push a code change up to

GitHub and watch in the AWS console with CodePipeline as each step

progresses and finally see the code pushed out to production. Here is a

screenshotshowingthefullprocessafteritisrun.Ifyouhappentohaveanerror

intheCodeBuildstep,thenyoucanopenthatupandseethedetailsandseehow

tofixtheerror.

Figure86–AWSCodePipelineusage

14.5MonitoringandAlerting

TheAWSElasticBeanstalkmanagementportalhasaMonitoringpagewith

whichyoucan viewkey machineperformance metrics.Youdefinitelywant to

opentheportalandlookatwhatisavailabletobemonitored.Notonlycanyou

lookat trending charts,but you canalsoset up alertsthat send you emails,or

even get text messages in the case of thresholds being crossed. Here is the

Monitoringpage:

Figure87-ElasticBeanstalkmonitoringpage

YoucanclickEditandselectwhatyouwanttobegraphedout.Youcanalso

clickthealarmbelliconandcreateanalarmthatwouldnotifyyouifyoucrossed

somethreshold.

Alternatively, you can open up the CloudWatch management console in

AWS and explore similar capabilities for metric viewing, logs, events and

alarms.

Figure88-AWSCloudWatchmanagementconsole

OnethingtoknowisthattheNode.jsprocesscancrashandifthathappens,

it needs to be restarted. In an IaaS environment, you would be responsible to

havesomemechanismtodetectthisandrestartyourNode.jsprocess.

There are NPM downloads such as “forever” and PM2 that do this. With

AWSElasticBeanstalk,itusesNginxandtherestartingishandledautomatically

foryou.ThisisanotherexampleofhowPaaSreallydonerightcanmakeyour

lifeeasier.

If your application is not available, reliable, and performant, then your

customersandyourbusinesswillsuffer.Thegoalofmonitoringandalertingis

tomaintainapplicationavailability,reliability,andperformance.Youdothisby

implementingApplicationPerformanceMonitoring(APM).Thisallowsyouto

discoverproblemsbeforeanyoneelsedoesandthentoachieveresolutionsinthe

leastamountoftime.

TobeginimplementinganAPMstrategy,youneedtoinstrumentyourcode

and surface events, logs, and metrics. Then you use an APM tool to chart out

performancemetricsandsetupalarms.

Youneedmetricssothatyouarenotflyingblind.Apilotcanflyaplanein

the dark because of instrumentation and telemetry. The last thing you want to

havetodoisremoteintoindividualserversandstartpokingaroundtosearchfor

a “needle in a haystack”. Instrumentation and monitoring is the only way you

willbeabletoscaleandsurvive.

AWSX-RayforcodeinstrumentationAPM

AWSoffersX-Rayasawaytoinstrumentyourcodeandhavethetelemetry

viewedinaportalthatletsyouinspecteachofthetracesthatyourapplicationis

processing.Atraceconsistsofinformationforwhatisoccurringinyourservice

layer.Forexample,youcanseeatraceforeveryHTTProuteendpointandeach

oftheverbsthatarebeinghandled.

To get X-Ray tracing to work with your Elastic Beanstalk application you

needtoturnonasetting.GointotheElasticBeanstalkconsoleandthenintothe

Configuration settings for the Software Configuration and check the box to

enable the X-Ray daemon. This will run the agent on the EC2 machines that

collect the data and forward it to the location it can be collected together for

viewingandalerting.ThiswillenableXRaytogetatmachineresources.

Note:NewRelicisatoolthathasbeenaroundforsometimeandisavery

comprehensive solution for gathering machine resource usage and transaction

instrumentation.LikeAWSX-Ray,youcanaddcodetoyourNode.jsprojectto

instrument the sending of telemetry. You can customize what is sent back and

addcustommetricsandevents.YoucanalsoaddascripttoyourHTMLclientto

instrumentfromtheclientsideallusage.NewRelichasarichcapabilitytoset

thresholdsandalertonthem.

TherearecostsassociatedwithusingX-Ray,butifyouhavetoprocessalot

oftracinginordertoincuranycost.PleaselookattheAWSdocumentationfor

anexplanationofthecost.Youcancontrolwhatitactuallycapturesonaroute-

by-route basis. You can completely ignore a route, or set it up to only have a

certainpercentageoftrafficcaptured.

X-Ray understands how to collect tracing information for transactions that

are handled by your Express usage in Node through the aws-xray-sdk npm

module.

Note: To download the module, you need to be running your machine

consoleasadmin.Thenyoucanrun“npminstallaws-xray-sdk–save”.

HereiscodetouseAWSX-RayinNodeJavaScript:

varAWSXRay=require('aws-xray-sdk');

app.use(AWSXRay.express.openSegment('NewsWatcher'));

//AllyourAPIroutes

app.use('/api/users',users);

app.use('/api/sessions',session);

app.use(AWSXRay.express.closeSegment());

IfyouthengotoX-RayintheAWSconsoleyoucanseethetraces.Youcan

alsosee all of the AWS resources in your service map. Here is what thetrace

viewlookslike.Youcanclickonatraceandseeitsdetails.

Figure89–X-Raytrace

If your code has an unhandled exception, then you could actually see the

stacktraceinthedetailsofthetrace.Youcanalsoaddextrainformationtoany

traceasfollows:

varsegment=AWSXRay.getSegment();

segment.addMetadata("errorHandler",err.toString());

IfyouwereusingtheAWSSDKinyourcode,youthenyoucanaddsome

codetomakeitawareofthosecalls,andthatusagewillbeinstrumented.Asof

yet,thereisnosupportforMongoDB,butIanticipatethattherewillbesoon.

Itispossibletoputincodetomeasureanypartofatransactiontracesuchas

someprocessingcodeyouwanttohaveafurtherbreakdownoftoseeitstiming

intheoveralltracetime.Thiscanbesynchronous,orasynchronouscode.

TheideaisthatthereisanincomingrequesttoyourNode.jsservice,suchas

aPOSTtoanAPIendpoint.AsingletraceIDisassignedtothatrequestandno

matterwhathappensuptothevery endofthatrequest,everysubpart ofwork

that goes on shares that ID. That means all of the work in your code, calls to

other backend HTTP services. Any use of other AWS resources is all tied

togetherandviewableunderthatsingleIDintheX-Rayconsole.

PARTIII:ThePresentationLayer

(React/HTML)

Partthreeofthisbookwillteachyouaboutthepresentationlayerofathree-

tierarchitecture.Indoingso,thesampleapplicationwillbeextendedtobringit

to a state where the UI is functional and interacting with the services layer.

Beforereadingthischapter,youshouldobtainabasicunderstandingofHTML.

In this part of the book, I present the technology of React as a framework

thatfitsnicelyintotheoverallarchitectureasawaytobinddatatoandfromthe

service layer web service into a UI. The UI will be rendered as an HTML

WebsiteandalsoasaNativemobileapplication.

If you have followed along in the previous chapters of this book, you will

realizethatthereisaservicelayerbuiltandtestedfortheUItoconnectto.This

is very important, as the Node.js application will perform the dual role of

servicingnotonlytheHTTP/RestAPI,butalsoofservingupyourReactHTML

andJavaScriptfilesetc.toabrowserasaSPAapplication.

Note:ItisnottheintentionofthisbooktobeacomprehensiveguidetoUX

designorSPAwebdesign.Iwillonlytouchonsomeconceptsandthensticktoa

narrowtechnology presentation.There are wholebooks eachdevoted toReact

andReactNative,so youcan understandthatwe canonlycover thebasics of

howeachworktogetyougoingonthistopic.

Chapter15:Fundamentals

I will now go over the fundamental concepts of the top tier in a three-tier

architecture. You will see what capabilities are essential and find a list of

questionsto consider when doing your code design. Youcan then get into the

specificsofReactasaJavaScriptUI frameworkandlearnhowitwillbe used

with the NewsWatcher sample application. One of the reasons that React was

chosen as a technology is because it fulfills the needs of this top layer of the

applicationarchitectureanddoessowiththeJavaScriptlanguage.

15.1DefinitionofthePresentationLayer

Anyapplicationthatrequiresuserinteractionwillneedapresentationlayer.

Humansneedapresentationlayertoviewdataandtoallowthemtoinputdata.

Forexample,anonlinebookstorewouldbepresentingalistofbookstoauserto

browsethrough.Eachbookoffered,mightcontainaphotoofthebookandthe

dataassociatedwithitsuchasthetitle,author,description,publicationdate,and

cost.Inputgatheredfromtheuserwouldbethingslikebookorders,reviews,and

customerservicequestions.

MV*andSPAdesigns

Youneed toemploy thetechniques ofabstraction andcomponentization in

all the layers of an architecture. This is no different with the coding in the

presentationlayer.

Oneofthebenefitsofchoosing aframeworkforyourpresentationlayeris

thatmostframeworksaresetuptoemploysometypeofMV*patternthatlends

itselftoanorganizedsetofcomponentsthatmakeupyourcode.

Make yourselves acquainted with MV* design patterns and with what a

SinglePageApplication(SPA)designentails.Thereareseveralexcellentbooks

available and plenty of online material to study. There are many things about

SPAdesignsthatmakethemagreatchoicetoday.

Note: You need to be aware that if you are concerned about SEO (Search

EngineOptimization)youmightneedtoswitchsomeofyourpagerenderingto

beserver-sideandnotbefullyrenderedasaclient-sideSPA.Thisissometimes

requiredforsearchenginestoindexyoursite.Reactcanbesetuptogiveyou

thebalanceyouneedforthis.Thereisashortchapterdevotedtohowtodothis.

Presentationlayerplanning

Therearemanydecisionsthatgointocreatingapresentationlayer.Thevery

firststepinvolvesplanningforwhattypeofinteractionsyouruserswouldwant

to take. You will want to make an initial sketch of the UI of your application

early on before coding anything up. This would happen concurrently as you

designtheservicelayer.Itiswisetopursueasimultaneousbottom-upandtop-

downapproach.

Knowingwhatoperationsandworkflowsareneededisanotherfirststepin

fleshingoutapresentationlayer.Thefollowingquestionsareusedtohelpyou

determinethedesignofapresentationlayer:

Have you done any of the following? sketching, prototyping,

storyboards, surveys, contextual inquiry, stakeholder interviews, A/B

testing,wireframes,sitemaps,personas,scenarios.Whatabouthuman

interaction,usability,andaccessibilitystudies?

Whatareyourdatasecurityandprivacyrequirements?

Howdopeoplesigninandbecomeauthorized?

Aretheremultiplestepsthatareprogressivelyrevealed,oneafter

another?

IsthereaneedforacustomizableUI?

Whatareyourglobalizationandlocalizationrequirements?

What devices are you targeting, such as desktop and mobile

platforms?

How can you keep data presentation to a minimum to not

overwhelmtheuser?

Whatformisdatabestpresentedin?

Whatisthebusinessneedthatcanbeaccomplished?

Whataccessibilityrequirementsarethere?

What are the navigation levels of the UI? Can you map out how

thenavigationworks?

Doyouneedauserfeedbackmechanism?

Doyouhaveofflinerequirements?

How is state stored in the client? Centralized across all view

hierarchy?

HowwilltheUIbedeployedandupdated?

Howwillusersenterdataandwhattestsareneededtovalidateit?

Can you map out the multi-step data entry forms and show the

branchingconditions?

Theanswerstothesequestionsshouldbecarefullydocumented.Beforeyou

roll anything out into your production environment, have experts reviewing

everything.

React is a great choice as a UI framework and fulfills all the needs of the

presentationlayeroftheapplicationarchitecture.ItusesJavaScript,soitfitsin

perfectly with the overall development stack we have been pursuing. You can

nowlearnthespecificsofReactandseeapracticalimplementationofhowitis

usedwiththeNewsWatchersampleapplication.

15.2IntroducingReact

ReactisaframeworkthatgivesyoutheabilitytodynamicallyrenderHTML

content in a browser web page. You write JavaScript to go alongside your

HTMLmarkup.TheJavaScriptcodecanleverageotheravailablelibrariestodo

things like HTTP/Rest requests, data binding and navigation and much more.

TheoverallcapabilitiesofReactprovidethemechanismstoenableyoutobuild

aSPA,atraditionalserver-siderenderedwebsite,orahybrid.

There have been other similar frameworks that have appeared at the same

timeasReactsuchasKnockout,js,Ember.js,Angular,Vue.jsandBackbone.js.

Reactismychoiceinthisbookasithasmanythingsgoingforit,suchasbeing

widelyadopted,andalsobecauseitisofficiallysponsoredbyalargecompany

(Facebook) as an open-source project. You can be up and running on the

desktop, mobile web, and even produce native mobile applications in a short

amountoftime.

Note:You see me referring to React as a framework.I do so in the larger

sense.React,alongwithyourchosenadditionsforotherfunctionality,providea

framework that you built your application presentation layer with. That is the

point of a framework, it is extensible. It just so happens that React is not as

prescriptive and feature capable as something like Angular. It is up to you to

combineitwiththeotherlibrariesofyourchoicetogiveyouallofthefeatures

youneed.

Reactfreesyoufromsomeofthelaboriouscodeyouusedtohavetowriteto

do DOM (Domain Object Model) manipulation. Because of its ability to bind

andaffectDOMelements,younolongerneedtoutilizelibrariessuchasjQuery.

ReactmakesHTMLdynamicsothatdatavaluesflowbackandforthforyou.

To use React, you write component files that render to the DOM. Be aware,

however,thatReactitselfdoesnotprovidecontrolelementsorstylingforyou,

asthatislefttotheHTMLmarkupandotherlibrariessuchasmaterialdesign,

Bootstrapandmanymorethatareavailable.

TheReactlibraryisconsumedbyplacingascriptinsideyourHTMLfilethat

pulls in React as JavaScript to execute from a CDN. You can also locally

bundledReactthroughWebpackandthenpullitinthatway.

Therearethird-partytoolsthatactasUIdesignstudios,whereyoucandrag

and drop elements and play around with them and have React code generated

andedited.Thisishandytobeabletovisualizeacomponentinanisolatedway

soastosaveyoutime.Otherwise,youneedtoseethecomponentinanoverall

application.LookintotoolssuchasStorybookandReactStyleguidist.

Note:Youmightrecallthatthe Node.jsExpress modulehasthe conceptof

serving up templates of “HTML-like” files and binding data to them on the

server-side,sothattheyarriveontheclientsideallfilledout.Expresssupports

many template formats such as Jade, EJS, mustache, and handlebars. I don’t

recommendthisserver-sidedatabindingtechnique.Instead,Igivepreferenceto

simplyservingtheHTMLthatworksasaSPAapplication,withtheclientside

usingReacttorequestdatathroughaRestAPIwithJSONandthendoingthe

binding on the client side. This alleviates the back-and-forth HTML page

requests. With a SPA, all of the navigation and page rendering is done on the

clientside,notontheserverside.Thisissimilartowhatyouwouldneedtodoif

youweretodevelopanativemobileapplication.YoucanalsostickwithHTML

insteadofhavingtolearnanewtemplatemarkupsyntaxsuchasJade.

15.3ReactwithonlyanHTMLfile

Youcould pullin the Reactlibrary witha scriptmarkup inan HTML file.

Thisisnotagoodidea,butIshowitasawaytounderstandwhatultimatelyhas

to happen to bring any code into an HTML page. React is just a library like

anythingelsethatrunsasascriptinabrowserthatexistsinanHTMLrendered

page.Youcanplacethistextinafileandopenitwithyourbrowserandseethat

itworks.

<!DOCTYPEhtml>

<html>

<head>

<metacharset="UTF-8"/>

<title>HelloWorld</title>

<script

src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.2/react.js">

</script>

<script

src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.2/react-dom.js">

</script>

<script

src="https://cdnjs.cloudflare.com/ajax/libs/babel-standalone/6.21.1/babel.min.js">

</script>

</head>

<body>

<divid="root"></div>

<scripttype="text/babel">

ReactDOM.render(

<h1>Hello,world!</h1>,

document.getElementById('root')

);

</script>

</body>

</html>

What you want to do, is install the necessary tooling on your developer

machinethatcanbeusedtodoabuildofyourcodetoproduceabundledfile

thathaseverythinginitfromyourprojecttouse.ThismeansReactisbundled

withyourwebsiteuploadofstaticfiles.YouendupwithageneratedHTMLfile

thatlookssomethinglikethefollowing:

<!DOCTYPEhtml>

<html>

<head>

<metacharset="UTF-8"/>

<title>HelloWorld</title>

</head>

<body>

<divid="root"></div>

<scripttype="text/javascript"

src="/static/js/main.35d639b7.js"></script>

</body>

</html>

The JavaScript file that you see being pulled in with the script tag is

generated in your build process on your machine and contains all the needed

ReactcodeandyourowncodethatisnecessarytoloadandrunyourSPApage.

Thenextsectionwillwalkyouthroughhowtoarriveatthefileshownabovein

thesimplestwaypossible.

15.4InstallationandAppCreation

If you really want to go through all of the work of installing each of the

modulesandtoolsfordoingreactdevelopmentyoucertainlycandothat.Todo

so, you can go to the official React website and follow their downloading

instructions.Youcanstartaprojectonyourownanddonpminstallsofmodules

such as react and react-dom. Then you would need to install tools for cross-

compilingandbundling.YouwouldwanttogettoolsinstalledfromNPMsuch

aswebpack,Babel,Autoprefixer,ESLint,Jest,andothertools.Youwouldneed

to become very familiar with the following terms and how to download and

managetoolsforeach.

cross-compiling:AllowsyoutousethelatestJavaScriptsyntaxandhaveit

turnedintoJavaScriptthatcanruninolderbrowsers.

bundling: This is how to gather all the files together for much better

performance and a controllable loading capability. Otherwise, you have to

reference the minified JavaScript files in your HTML and take the hit on the

initialpageload.Goingwithbundlingalsogetsyoupreparedforbeingableto

deploy as a native application on a mobile device, as that is the only way to

accomplishthat.

Insteadofinstalling thetoolingon yourown, youcansimply installatool

that populates a React project for you with some example code and all of the

tooling.

Usingcreate-react-app

The task of getting the initial project files in place to have the traditional

“Helloworld”applicationrunningmayseemalittledauntingatfirst.Thebest

approach is to install and use the create-react-app utility. Install this tool as

follows:

npminstall-gcreate-react-app

Youcouldnowrunthisandyouwouldhaveanapplicationallreadytorun.

Inourcase,wealreadyhaveanexistingfolderwithaNode.jsapplicationinit.

WhatwewilldoiscreatetheReactapplicationinaseparatefolder.Thensimply

copy over the necessary folders from the created project over into the Node

project. This is the best approach to take for now. Run the following on the

commandlineinacompletelynewdirectory.ThiswillcreateyourbasicReact

applicationthatyoucanpullfrom:

create-react-appmy-app

Copyoverthe“public”and“src”directories.Lookinthepackage.jsonand

the.gitignorefilesandbringoverwhatismissingintheonesintheNode.jsfiles.

Whileyouareeditingthe.gitignorefile,add“.chrome”asaline,asthiswillbe

neededlater.Forthepackage.jsonfilethatmeantjustbringingoverafewofthe

scriptcommands.Irenamedthem,soasnottoconflictwithanythingIhadfor

myNode.jsproject.

"start-react":"react-scriptsstart",

"build-react":"react-scriptsbuild",

"test-react":"react-scriptstest--env=jsdom"

You also need to copy the Node.js package.json lines that were for the

following dependencies: react, react-dom and react-scripts. Then you can run

npminstallandhavethosereadytouse.

To run the Reactapplication, you cannow run the command“react-scripts

start”andthatwouldbeservedupforyouonyourlocalmachine.Wehaveputa

script command in package.json for these. You can thus run “npm run start-

react”. If you run that, you will see the following launched in the Chrome

browser:

Figure90-StarterprojectscreenforReactinstall

Note:Node.jshasitsown commandstogetaninitial applicationstructure

set up and configured on your development machine. It can be confusing on

whattodo foryourinitialprojectcreation,asyou havejust seenthatthere is

alsoacommandlineutilitytocreateaReactapplication.Thereisalotmoreto

aReactinstall,however,especiallyifyouuseaframeworksetup,likethecreate-

react-apputility.ThusIhadtoessentiallygeteachprojectsetupindependently

andthenbringoneovertotheother.Thereareotherutilitiesouttheresuchas

Yeomanthatcangiveyouthecompletescaffoldingforeverythinginonesingle

initialization.Thereisalsoageneratornamedreact-fullstackthatcanbeusedas

astartingpointforyourproject.Ofcourse,youcanalsocloneanyone’sGitHub

application,suchasmine,andhaveeverythingallreadytouse.

ServingupReactfromNode.jsandExpress

Before getting into the heavy coding in React, make sure that the initial

React page is served up correctly from the Node.js service. The details of

making your Node.js Express application serve up the React index.html and

JavaScriptcodeasaWebPageareextremelysimple.Herearethebareminimum

linesofcodeyouneedtodothat.

constexpress=require('express');

constpath=require('path');

constapp=express();

app.get('/',function(req,res){

res.sendFile(path.join(__dirname,'build','index.html'));

});

app.use(express.static(path.join(__dirname,'build')));

app.listen(3000);

Inthenextchapter,youwillseethedetailsoftheNewsWatcherapplication

code for the presentation layer. You will see how these lines are incorporated

intotheserver.jsfile.TorunitwithNode,youputinthoselinesshownabove

andthenneedtofirstdoabuildandbundleoftheReactcodebeforetheNode

servicecanbestarteduptoserveit.TodothebuildandbundleofReact,yourun

thefollowing:

npmrunbuild-react

YoucouldthenpressF5inVisualStudioCodeandthenyourapplicationis

once again being served up, this time through Node, and you can go to

http://localhost:3000/ in your browser to see it. If you are curious, you might

wanttoinspectthebuilddirectorythatiscreatedandlookateachofthefilesto

seeexactlywhatisbeingservedandrunontheclientbrowser.

Theindex.htmltemplateandgeneratedone

If you recall, I showed some HTML that was all self-contained. It had

everythingneededtorunaReactbackedwebpage.Ithadthescriptsnecessary

tousethe Reactlibraries. However,withthe projectstructurecreated withthe

create-react-app utility, it works completely differently. You still have the

index.html file. That is used as the starting point for the one being generated.

Youwillnoticethatitdoesnothaveanyscriptspulledin.Itlooksasfollows:

<!doctypehtml>

<htmllang="en">

<head>

<metacharset="utf-8">

<metaname="viewport"content="width=device-width,initial-scale=1">

<linkrel="shortcuticon"href="%PUBLIC_URL%/favicon.ico">

<title>ReactApp</title>

</head>

<body>

<divid="root"></div>

</body>

</html>

ThisHTMLfileisservingasapointtemplatefileandhasthatsamedivwith

anidof“root”.Bytemplate,Imean,itistakenatbuildtimeandalteredandthe

finalindex.htmlfileiscreatedandplacedinthebuilddirectory.Hereiswhatthat

lookslike:

<!DOCTYPEhtml>

<htmllang="en">

<head>

<metacharset="utf-8">

to-fit=no">

<metaname="theme-color"content="#000000">

<linkrel="manifest"href="/manifest.json">

<linkrel="shortcuticon"href="/favicon.ico">

<title>ReactApp</title>

<linkhref="/static/css/main.c17080f1.css"rel="stylesheet">

</head>

<body>

<divid="root"></div>

<scripttype="text/javascript"

src="/static/js/main.35d639b7.js"></script>

</body>

</html>

Thestartingpointforyourcode

Wecanfinallygettotherealpointyouhavebeenwaitingfor.Youcannow

learnwhatthestartingpointisforwhereyouplaceyourowncustomcodefor

your React application. This code belongs in the index.js file found in the src

folder.ThisiswhatisusedatbuildtimetobethestartingpointofyourReact

app.HereisthefilethattheReactbuildtoolsknowaboutandusetobootstrap

yourapplication:

//index.js

importReactfrom'react';

importReactDOMfrom'react-dom';

importAppfrom'./App';

import'./index.css';

ReactDOM.render(

<App/>,

document.getElementById('root')

);

You see the same ReactDOM.render() call that was in the initial self-

containedHTMLfile.Itisjustsplitoutandusedatbuildtimetobepulledin.

Whatyouseehereisthattheindex.jsfileisimportingtheApp.jsfile.Inthere

youwillfindthedefinitionofthecomponentnamedAppthatisbeingrendered.

Reactcodeis basedontheconceptof componentsthatare assembledto build

theUI.Eventually,itallleadsdowntoHTML.

Theindex.jsfileisusedbyReacttodotheinitialelementinsertionintothe

divthathadthe“root”id.ThiscodeusestheApp.jsfileasthestartingpointfor

whereyouplacealloftheJavaScriptandHTMLtagstodotherenderingofwhat

yourUIwillbe.Hereiswhattheinitialcontentsarefortherenderingasfoundin

theApp.jsfilewhenwecreateditwiththecreate-react-appcommand:

//App.js

importReact,{Component}from'react';

importlogofrom'./logo.svg';

import'./App.css';

classAppextendsComponent{

render(){

return(

<divclassName="App">

<headerclassName="App-header">

<imgsrc={logo}className="App-logo"alt="logo"/>

<h1className="App-title">WelcometoReact</h1>

</header>

<pclassName="App-intro">

Togetstarted,edit<code>src/App.js</code>andsavetoreload.

</p>

</div>

);

}

exportdefaultApp;

Whenyourun“npmrunbuild-react”fromthecommandline,thisbuildsand

bundleseverythingforusagetobe servedupfromyour Node.jsservice.After

thebuildcommandcompletes,youwillfindthatyouhaveanewfoldernamed

“build”.Itisinherethatthefilesareplacedthatarefullycapableofbeingused

aspagestobeloadedandseeninabrowser.Forexample,youwillnowfinda

newindex.htmlfileinthebuildfolderthatwasshownalready.

Thebigdifferencenowbetweenthisfileandtheoneinyourpublicfolderis

there is a style sheet and script file brought in that are used to run the React

application. This is why a Node.js server can be set up to serve up the build

folder index.html file and also how it can get at its own built files for all the

ReactandprovidedJavaScriptfilestorun.

DebuggingReactcode

Youwillwant to know how to debugthis new React code. Debugging the

Node.jscodewassimple.YoujustplacedabreakpointinyourNodeJavaScript

codeandyoustoponitwhenthatisexecutedandstepthroughit.Section18.3

willcontainmoreonthistopic.

ForReactcode,itisreallyeasytodebugyourcodeintheChromebrowser.

YoucangettothecodeandsetbreakpointsbyselectingfromtheChromemenu

More tools->Developertools. You can then be in an environment where you

canalsoinspecttheDOMandlookatconsoleoutputandmuchmore.

You will need to run Node separately now, either from the command line

(“npm start”), or in VS Code you can press F5. Then you can go to

http://localhost:3000/inChromeandwiththedevelopertools,debugtheReact

code.Youcan,atthesametime,debugserver-sideNodecodeinVSCodeasitis

running.

YoucanlookuponlinetofindouthowtousetheChromeDevelopertools.

Tosetabreakpoint,youjustopentheSourcestabandthenontheleftsideyou

willfindthefolderswiththecodeinit.Gotothestaticfolderandinthere,will

bethejsfolder.Opentheviewsfolderandselectafilesuchasloginview.jsand

youcansetbreakpointsandstepthroughthecode.

Thenicethingthatishappening,isthatyouarelookingatyourcodebefore

itwascross-compiledandbundled.Therearemappingfilescreatedbehindthe

scenesthatallowtheactualcodethatisbeingruntobemappedtotheoriginal

lines for your ease in debugging it all. Here is what it looks like with

NewWatcherrunningandhittingabreakpointinthecode.

Figure91-ChromeDevelopertools

15.5TheBasicsofReactrenderingwith

Components

PerhapsyouhaveheardthatReactdoesnotuseHTMLfileslikeatraditional

WebServermightserveup.Reactinsteadusescodefilestocreatecomponents

thatgetpiecedtogethertocreateyourUI.ItisalldonethroughstandardDOM

manipulationcode atthe lowest level.Don’tbe thinkingyou are gettingaway

from HTML markup and CSS styling by any means. There will ultimately be

justasmuchofthatrequiredtocreateyourapplication.

WithalltheReactcomponentabstractionsyoupiecetogether,atthelowest

levelyouwillendupprovidingHTMLmarkupthatgetsrenderedtotheDOM.

YouhaveallthestandardsetofHTMLelementstopullfrom.Youwillusetags

suchas<p>,<a>,<ul>,<button>,<input>,<iframe>,<img>etc.Youcanalso

addcontent,attributesandstyling.

ReacthasyouprovideHTML,butyoudoitinaJavaScriptfile(.jsor.jsx)

instead.ThecodeisJavaScriptthatallowscertainadditionalsyntax.Thesefiles

arecross-compiledintotheactualJavaScriptcodethataBrowsercanrun.

Atthe startof thischapter,you learnedthat thereis a top-levelindex.html

filethathasadivelementinitasfollows(seethelineinboldtype):

<body>

<divid="root"></div>

<scripttype="text/javascript"src="/static/js/main.3dd1fbc9.js"></script>

</body>

IfyouunderstandalittleaboutbrowserDOMmanipulationAPIs,youknow

thatyoucancreateanythingyouwantintheDOMthroughcode.Forexample,

you might insert a text element using JavaScript code. Furthermore, you can

referenceanyexistingelementbyanIDyougiveit.Inthatway,youcanfindit

in code and alter it. This means you could find a div and add a child HTML

elementtoit.ThatchildelementwouldthenshowupintherenderedUIaswell.

Hereis some plain HTML code with no React usage at all. Youcan place

thisin afile and openit in yourbrowser totry it out.It uses someJavaScript

code inside a script tag. The standard document object available in browser

JavaScript is used to access your web page. It is part of an agreed-upon API

standardavailableinanybrowser.Thecodecreatesatextelementasachildof

thedivelement.

<!DOCTYPEhtml>

<body>

<divid="root"></div>

</body>

vartextnode=document.createTextNode("WelcometoReact");

document.getElementById("root").appendChild(textnode);

</script>

</html>

React employs this exact mechanism to take anything you want to be

rendered and presented on the page. To begin with, React is rendering to the

maindivyousawabovewiththeidof“root”.Thereisastartingpieceofcodein

afilenamedindex.jsthatisusedasthepagegetsrendered.Atthetimetheinitial

Reactapplicationwascreated,itlookedasfollows:

//index.js

importReactfrom'react';

importReactDOMfrom'react-dom';

importAppfrom'./App';

import'./index.css';

ReactDOM.render(<App/>,document.getElementById('root'));

ThiscodeeventuallygetsbuiltandmorphedintoanotherJavaScriptfilethat

utilizestheReactAPItomakecallsthatusethedocumentobjectanddothings

likecallappendChild.ThisisthefirstpieceofcodethattheReactlibraryusesto

rendersomething.

ThecallyouseetoReactDOM.render(…)isacceptingasthefirstargument

whatelementyouwanttorender.Appisacomponentfromanotherfilethatwas

imported. The second argument to the render() call is telling React where to

rendertheUI.Thatiswherewearereferringtothe“root”divintheindex.html

file.

ThelastfiletolookatintheprocessofunderstandinghowReactdoesDOM

rendering,isthefilethatcontainsourcustomdefinedcomponent.Inthiscase,

thefileisnamedApp.js.Thisthenisthepatternforthecodethatyouwillcreate

over and over as you define the UI you want to be rendered. Here are the

contentsofthatfile:

//App.js

importReact,{Component}from'react';

importlogofrom'./logo.svg';

import'./App.css';

classAppextendsComponent{

render(){

return(

<divclassName="App">

<divclassName="App-header">

<imgsrc={logo}className="App-logo"alt="logo"/>

<h2>WelcometoReact</h2>

</div>

<p>edit<code>src/App.js</code>toreload.</p>

</div>

);

}

exportdefaultApp;

ReactisaJavaScriptlibrarythatexposesanAPI.PartofthatAPIisaclass

youcanderivefromthatitunderstands.Componentisaclassyouderivefrom

and write code to contain your own UI to render. App is your highest parent

controlandisderivedfromComponent.Youwillcreateacompletehierarchyof

othercomponentsyoudefineandmakeuseof.Intheapp.jsfile,youexportthe

Appcomponentsothatitcanbeimportedandusedintheindex.jsfile.

Noticethefirstlineofapp.jsthatdoesanimportofReact.Thismustpresent

even though you don’t see the usage of that object. This is because the

JavaScriptcodeistransformedbyBabel,andwillbeusedinthatcode.

Note:Byconvention,youstartyourcustomcomponentswithanuppercase

letter.ThisthenhelpsyoudistinguishthesefromHTMLelementsthatarealways

alllowercase(i.e.div).

TherenderMethod

The Component class requires you to provide a method named “render”.

ReacttakesyourAppcomponentandcallstherender()methodtogetanything

thatneedstoberendered.BeawarethatthisisnotactuallyaJavaScriptfilethat

canberun.Asexplained,itisusedintheReactbuildprocessandturneditinto

something that can be run. You may have seen that the return in the render

methodisnotreallyreturninganythingacceptableinJavaScript.

ThisuseofJavaScriptiswhatisreferredtoastheJSXsyntax.Itallowsyou

tomixHTMLtagsandJavaScriptrightinonefile.Forexample,youcanhave

conditionalcode that determineswhat elementsshould be rendered on apage.

Lookatthefollowingexample:

render(){

if(user.account.total>1000){

return(<h1>Youmaintainedtheminimumrequiredbalance!</h1>);

}

return(<h1>Youarebelowyourminimumrequiredbalance</h1>);

}

BeawarethatyoucanonlyreturnonesingleHTMLelementfromtherender

function.Meaning,thereneedstobeasingleparentelementsuchasadiv.For

example, you can’t return two h1 elements unless they are wrapped in a div

element.Youcanreturnasingleh1elementasshown.

There are a few subtle differences in the HTML as it is used with React.

Theseareasfollows:

The React DOM property naming convention is to use camel

casing and has altered some of the standard properties slightly. For

example,tabindex,becomestabIndexandclassbecomesclassName.

You use curly braces to embed JavaScript expressions in an

attribute. For example, specifying the source image for an <img>

element,youwouldhavetodothis:src={myObject.someImageURL}.

Youcanuseanyobjectinyourcodethisway.

To summarize, you can understand that a React application consists of

providedComponentsthatcontainHTMLtoberendered.Thisthenistakenby

theReactAPI,andusingthedocumentobject,Reactrendersthatintotheactual

browserDOM.

Note: The React API also has calls for creating individual elements. For

example, there is a method React.createElement() that takes arguments

specifyingtheelementtag,attributesandcontent.ThisisactuallywhattheJSX

getsturnedintointhebuildprocesswheretheBabelcompilerisrun.Thisbook

willstickwithusingComponentsinJSXfilesyntax.

MoreaboutmixingJavaScriptwithmarkup

YoubrieflysawthatinJSXsyntax,youcanhaveconditionalcodethatuses

controlflowofJavaScripttodecidewhatmarkuptoreturnforrendering.Thisis

reallyoneofthegreatadvantagesofusingReact.

Imagineyouhadsomebackendservicethatyoucalltoretrievesomedata,

suchasfromanHTTP/Restwebservice.Youcantakethatdataandthenprocess

itintheJSXcodeandproducewhateverUIyoulike.

Thenext examplecode willshow the takingof some datathat showshow

codecanmakeuseofit.Thedataisalistofcommentstobedisplayed.Thecode

will first determine an appropriate piece of text to display and will then loop

through the comments and display each using another lower level component.

TheloopingcodemakesuseoftheJavaScriptmap()functionofthearrayobject

toloopthroughandgeneratetheHTML.

The CommentListItem is some custom component you would provide that

wouldbereusableandknowhowtorenderanindividualcomment.

classCommentsextendsComponent{

render(){

constcoms=props.comments;

return(

{coms.length>0&&

<h2>Thereare{coms.length}comments</h2>

}

<ul>

{coms.map((comment,idx)=>

<CommentListItemkey={idx}

title={comment.title}/>

)}

</ul>

);

}

Note:YouhavethefullpowerofpullinginJavaScriptlibrarieslikelodash

thatcanbeusedtoproducelogictooutputthemarkupyouwantrendered.

You can see what is possible in code that combines JavaScript code with

HTMLtagsbeingreturned.JavaScriptcanbeinsertedinthemiddleofHTML

markupbyplacingitinsidecurlybraces.

Youcansetupblocksofcodethatrenderbasedonatest,suchastheoneyou

seetestingcoms.length.Finally,youseetheuseofthemap()functionthatlets

youloopthroughanditeratetorenderacomponent.Inthiscase,itisutilizing

anothercomponenttocreatethelielementsthat wouldbeplacedinsidetheul

elementtocontaineachlistitem.

Note:IfyoueverhavelogicthatneedstodecidebetweenshowingsomeUI

orhidingit,itisbettertonotuseCSStohidethetags,butinsteadtojustnotto

returnanythingatall.Yousimplyreturnnullfromthecodeandthatisit.

Alternatesyntaxusingafunction

This book will always show components being built that are derived from

Component. Theres is an alternate syntax that can be used that you should be

awareofincaseyoueverseeit.

Here is a component that is shown using the Component class syntax first

and then is shown using the alternate syntax. One is using the arrow function

syntax.Allareconsumedinthesamewayandhavethepropsavailabletopass

in.Withtheclassusage,thepropsareofcoursepartoftheclassinstance.

//CodeusingaComponentclass

classGreeting1extendsReact.Component{

render(){

return<h1>Greetings,{this.props.name}</h1>;

}

//Codeasafunction

functionGreeting2(props){

return<h1>Greetings,{props.name}</h1>;

}

//Witharrowfunctionsyntax

constGreeting3=({name})=>{

<h1>Greetings,{name}</h1>;

}

//Usage

constg1=<Greeting1name="Joe"/>;

constg2=<Greeting2name="Mary"/>;

constg3=<Greeting3name="Paul"/>;

Therearelimitationsifyouusethefunctionsyntax.Forexample,youcannot

use Redux. You will also learn about what lifecycle events are, and find that

theseareonlyavailableifyouusetheComponentclass.Youwillonlyseecode

examplesinthisbookusingtheReact.Componentclassextension.

15.6CustomComponentsandProps

YouhaveseenasimplecomponentnamedApp.Thiswasusedintheinitially

createdReactapplication.ItderivedfromtheReactComponentclassandhada

render()method.Anyuser-definedcomponentcanactasawrapperaroundthe

complexity of some underlying UI logic. Each component is really existing in

ordertorendersomeHTML.Youcreatecomponentstosplitfunctionalityoutin

awaythatcanbeself-containedandreusable.

Componentscanbeverycomplexandcanhavepropertiespassedintothem

tocontroldifferentaspectsoftheirrendering.Forexample,youcouldpassinan

arrayofdatathatthecomponentthenrenders.

Componentscanmaintainstate,suchasdataaboutwhatisgoingonwiththe

component. Passed in properties and internal state can be updated

asynchronously and the UI gets rendered when the changes happen. React

detectsthechanges,andupdateswillbebatchedtoprovidebetterperformance.

The capability of having properties on a component is what React calls

“props”.Acomponentcanhavemultiplepropsonit.Theirusagelooksjustlike

thatofattributeswhenyouuseacomponentinothermarkup.Forexample,you

couldpass insome textas aprop asshown inthe followingexample. We can

altertheindex.jsfiletolookasfollows:

ReactDOM.render(

<Appname="Joe"/>,

document.getElementById('root')

);

YoucanaltertheAppcomponentasfollowsandthenseeitwork.Hereisthe

altered App.js file to use the passed in name. The text is taken and displayed

alongwiththewelcometextthatspecifiesaperson’sname.Thelineinboldis

theonlymodificationneeded.

//App.js

importReact,{Component}from'react';

importlogofrom'./logo.svg';

import'./App.css';

classAppextendsComponent{

render(){

return(

<divclassName="App">

<divclassName="App-header">

<imgsrc={logo}className="App-logo"alt="logo"/>

<h2>WelcometoReact{this.props.name}</h2>

</div>

<pclassName="App-intro">

Togetstarted,edit<code>src/App.js</code>andsavetoreload.

</p>

</div>

);

}

exportdefaultApp;

Everypropshowsupinthecomponentaspropertiesonthepropsobject,and

are available in the component class via this.props. The name property is

accessedintheprecedingexampleasthis.props.name.

A property is only used as an input mechanism to a component. It is

importanttoknowthatPropsareimmutableinsidethecomponent.Forexample,

you could not have code that alters this.props.name. Props are meant to be

passed in and consumed by the Component when it renders itself. The calling

codeiswhatsetsthepropsandcanchangethemifneeded.Ifthecallingcode

were to have a timer that later changed the name prop, then the UI would re-

render.

Youwilllearnthattherearewaysintheparentusageofacomponentwhere

itcanbenotifiedofsomechangeofdataacquiredbyachildcomponent.Thisis

donebypassingacallbackfunctionasapropthatthechildcomponentcanuse.

This is totally up to you to define. There are predefined callbacks that can be

used as standard HTML attribute notification mechanisms, such as OnClick().

These,however,arenotsurfacedbackuptotheparent,unlessyoutakestepsto

providethemechanismtofeeditbackup.

15.7ComponentsandState

Aswasmentioned,propertiesarepassedintoacomponenttobeconsumed

via this.props. The other object available is this.state. State is internal to the

componentandcanbechangedinternally.Reactknowsaboutstatechangesand

canrefreshthe UItoreflect anychanges.Look atthefollowing codeand you

willfindtheusethethis.stateobject.Hereisalsowhereyouwillseetheusageof

aconstructorforyourcomponentclass.

classAppextendsComponent{

constructor(props){

super(props);

this.state={name:this.props.name};

}

render(){

setTimeout(()=>{this.setState(currState=>({

name:"John"

}))},5000);

return(

<div>

<h2>WelcometoReact{this.state.name}</h2>

</div>

);

}

It is only in the constructor that you can make any direct assignment to

this.state.Inotherplacesinyourcode,youcallthis.setState()tomakechanges.

Also note that you call super(), which is the Component base class call for

initializingitsconstructor.Youcanseeinthecodethattheinitialnameproperty

issetfromthepropsthatarepassedinandthenalteredinatimercallback.

Anytime you alter the state outside of the constructor, you must use the

this.setState().You only need to provide the properties you want to refresh in

each case. It won’t affect any of the other state properties of the component.

Reactwilldo a UIrefresh whenit sees statechanges havehappened andmay

batchupseveralbeforeitdoesaUIrefresh.

Withtheusageofthis.setState()youcanaccessthecurrentstatebeforeyou

makeyourchange.Inthepreviousexample,wedon’tactuallyusethatcapability

intheexample.

The state should be moved up to the highest component which has it in

commonacrosschildcomponents.Aparentcanhaveastatepropertyofitsown

andpassthatintothechildcomponentasaprop.Whentheparentstatechanges,

thechildwillseetheupdateandReactwillre-renderthecomponentifnecessary.

Model,ViewandController

IfyouwanttothinkintermsofhowanMVCarchitectureworks, youcan

recognize that React fits into this model by providing both the View and the

Controller capability. The View is what renders DOM elements from your

Component render method. The Controller is the rest of the logic that acts on

behalfofthepassedinpropsandmanagedstate.

YourReactcomponentcandothefetchingoftheModeldata.Youhavecode

fetchthedataandpassitaspropstoacomponent,orinthecomponentitselfyou

fetchitandsetthestate.YoucanutilizelibrariessuchasFlux/Reduxtodothe

coordination of data fetching and the setting of state in the code of the

component. Here is an image that might help you visualize how this all fits

togetherinanMVCdesign:

Figure92-ReactMVCconcepts

Note:Youcandebateexactlywheretokeepthecontrollerandtheviewcode.

Youcouldhaveitinonesinglecomponent.Itmightbebest,however,toseparate

the controller code from the view code. To do this you could have a data

componentthatusesadatastoreandthenthedatacomponentwouldpassprops

to a view only presentation component. These two types of components are

called Container and Presentation Components (also called "smart" and

"dumb").MorewillbesaidaboutthislaterinthesectiononReduxusage.

15.8EventHandlers

TheHTMLtagsusedinrenderingcanhaveeventhandlersgiventothemas

appropriate.Thiswouldbeforeventssuchasaclickorkeypress.InReactJSX

files, this means you provide a JavaScript function in your React Component

class and then use curly braces for specifying the event handler. Events are

specifiedwithcamelcasecharacters.

Hereisanexamplethataddsahandlerforaclickeventonabutton.Notice

thearrowfunctionisnecessarytohavethecorrectcontextfor“this”.Otherwise,

thethis.setState()wouldhaveanerrorthrown.

classAppextendsComponent{

constructor(props){

super(props);

this.state={count:0};

}

handleClick=(e)=>{

this.setState(currState=>({count:currState.count+1}));

}

render(){

return(

<div>

<h1>Youhaveclickedthebutton{this.state.count}times</h1>

<buttononClick={this.handleClick}>

Clickit

</button>

</div>

);

}

The “e” function argument is actually an event object that has a lot of

propertiesandmethodsonittobeusedasneeded.Forexample,youmightwant

tocaptureacertainkeythatwaspressedandthenpreventitfrompropagating.

Therearecaseswhereyoudon’twantaclick,orkeypresstobeprocessedany

further.Topreventpropagationofanevent,youwouldcalle.preventDefault().

PassingupEventHandling

Youcanhavetheconsumerofacustomcomponentbenotifiedofeventsina

sub-componentbyexposingthemasproperties.Ifyouhavemultipleevents,you

needtonameeacheventpropsomethingdifferent.Inthecodebelow,MyButton

just calls the prop callback method in its onClick event. In this way, the App

componentcanhavethecodethatrespondstothebuttonclick.

classAppextendsComponent{

constructor(props){

super(props);

this.state={count:0};

//Alternatewaytoensurecorrectthisbinding

this.handleClick=this.handleClick.bind(this);

}

handleClick(e){

this.setState(currState=>({count:currState.count+1}));

}

render(){

return(

<div>

<h1>Youhaveclickedthebutton{this.state.count}times</h1>

<MyButtonmyClickHandler={this.handleClick}/>

</div>

);

}

classMyButtonextendsComponent{

render(){

return(

<div>

<buttononClick={this.props.myClickHandler}>

Clickit

</button>

</div>

);

}

The code above also shows the alternate syntax for binding to have the

correctcallingonthecallback,otherwise,youcanusethearrowfunctionsyntax.

15.9ComponentContainment

There is a special props property that is always available on a component

thatisused forpassing inchildcomponentsorchild HTMLelements directly.

Thenameofthepropertyis“children”.Thismeansthatyoucanenclosechildren

intheusageofacustomcomponenttag,andthengetaccesstothem.Hereisthe

previous example, but this time, the text of the button is passed as part of the

children.

classAppextendsComponent{

constructor(props){

super(props);

this.state={count:0};

}

handleClick=(e)=>{

this.setState(currState=>({

count:currState.count+1

}));

}

render(){

return(

<div>

<h1>Youhaveclickedthebutton{this.state.count}times</h1>

<MyButtonmyClickHandler={this.handleClick}>

<h1>Clickit</h1>

</MyButton>

</div>

);

}

classMyButtonextendsComponent{

render(){

return(

<div>

<buttononClick={this.props.myClickHandler}>

{this.props.children}

</button>

</div>

);

}

The previous example shows an HTML tag being enclosed as part of the

children.YoucanalsohaveotherJSXcomponentsorstringliteralsbechildren

aswell.

Note:WhentheJSXistakenandrendered,anywhitespaceatthebeginning

andendofalineisremoved,aswellasblanklines.Newlinesthatoccurinthe

middleofstringliteralsarecondenseddowntoasinglespacecharacter.

You can always pass in any name for a property that contains multiple

elements. This may become necessary if you have multiple sub-components

being used in elements and are displaying them. For example, in the previous

examplethebuttontextcouldhavebeenpassedinasfollows:

classMyButtonextendsComponent{

render(){

return(

<div>

<buttononClick={this.props.myClickHandler}>

{this.props.myButtonText}

</button>

</div>

);

}

15.10HTMLForms

Reactsupports the creation of HTML forms via the standard formtag that

existsinHTML.Youcancreateacomponentthatrendersitselfandprovidesthe

formtagandsupportstheeventhandlingfunctionforthesubmission.Asimple

exampleisasfollow:

classAppextendsReact.Component{

constructor(props){

super(props);

this.state={value:''};

}

handleChange=(event)=>{

this.setState({value:event.target.value});

}

handleSubmit=(event)=>{

alert('Thetestsubmittedis:'+this.state.value);

event.preventDefault();

}

render(){

return(

<formonSubmit={this.handleSubmit}>

<h2>Textis{this.state.value}</h2>

<label>

Name:

<inputtype="text"onChange={this.handleChange}/>

</label>

<inputtype="submit"value="Submit"/>

</form>

);

}

Notonlydoestheabovecodehavethesubmissionhandling,butyoualsosee

codetohandlethebindingforchangesofaninputelementsothatthechanges

are rendered. You can also get events on changes for a checkbox, list, select

controlandothers.TheonChangepropertyisusedforthoseevents.

15.11LifecycleofaComponent

React sets up a series of stages that a component goes through as it gets

renderedandeventuallydestroyedandtakenoutoftheDOM.Youcanprovide

methodsonyourclassthatgetcalledwhenthecomponententerseachofthese

stages.

These methods are called “lifecycle hooks”. For example, the

componentDidMount() method is called after the component output has been

rendered to the DOM. You can then know that your component is visible and

take any action you need. You might want to create a timer that refreshes the

state of the component every few seconds, if data changes in the backend

serviceslayermighthavehappened.

AnothermethodavailableiscomponentWillUnmount(),whichiscalledjust

beforeacomponentistakenoutoftheDOM.

15.12TypecheckingyourProps

Youcanmakeuse ofthepropTypespropertyofyourcomponentto setthe

expected property types of the usage of your component. This way, you can

catcherrorsinitsusage.Thisisonlydonewhenrunningindevelopmentmode.

You can test for individual props such as an object, array, func, string,

numberandmanyothertypes.Youcanalsospecifyifagivenpropisrequiredor

not. There is also a means of specifying default values for props that are not

passedinbyaparent.Hereisanexamplethatshowsoffthesecapabilities:

importPropTypesfrom'prop-types';

classAppextendsComponent{

render(){

return(

<divclassName="App">

<divclassName="App-header">

<h2>WelcometoReact{this.props.name1}</h2>

<h2>WelcometoReact{this.props.name2}</h2>

</div>

);

}

App.propTypes={

name1:PropTypes.string.isRequired,

name2:PropTypes.string

};

App.defaultProps={

name2:'Unknownperson'

};

15.13GettingareferencetoaDOM

element

There may be cases where you need to call the actual DOM manipulation

functionsonanelement.Todothat,youneedtogetareferencetobeabletouse

it. React has a special syntax that you can employ to set the reference to the

element.Hereiswhatthatlookslike.Youwoulddothisifyoucan’tstoreaway

areferenceinyourcomponentclasstoaninputelement.

<inputtype="text"ref={(input)=>{this.textInput=input;}}/>

Insomeothercode,youcouldmakeacalltousethis.textInputtogetorset

its value. There is also a function provided in the React library named

findDOMNode that can be used to find any element in code. I prefer to find

waysaroundusingeitherofthesemechanismsastheyreallyaretherejustforthe

oddcase,asthereusuallyisawaytofindanycomponentincontext,andgetits

valueinsimplerways.

Chapter16:FurtherTopics

ThataboutcoversthebasicsofusingReact.Thischapterwillcoverfurther

topicsthatyouwillwanttoknowtoachieveafullrobustapplication.

16.1UsingReactRouter

Itisassumedthatawebsitewillhavemanydifferentpagestobeviewed.Ina

SPA,thismeansmanydifferentviewswillberenderedasneeded.Toaccomplish

thisyouwillneedtorendersometypeofnavigationbarwithlinkstoclickon.

Fromthoselinkstheviewwouldbechanged.

React does not come with any built-in capability for handling page view

routing. You could certainly create your own code to render some type of

navigation bar across the top of your main site page and then do the work to

changetodifferentpageviewsinsideofthat.

ThesolutionIwillgiveinthisbookistoutilizeapopularopensourcenpm

packagenamedReactRouter.Thereareactuallytwodifferentpackagesyoucan

findthatarecreatedbythesamegroup ofpeople.Oneisforwebsites andthe

otherisfornativeappdevelopment.Wewillusethewebsiteversion,whichis

foundhere-https://www.npmjs.com/package/react-router-dom.Incaseyouare

interested, the native app version is named react-router-native. React Router

givesyoutheabilitytocreateadynamicnavigationexperienceonyoursite.To

getstarted,youcaninstallthepackageintoyourprojectasfollows:

npminstall--savereact-router-dom

Note:ReactRoutercanbeutilizedontheserver-sidetoreturnpagesthatare

renderedwithdata,orReactRoutercanbeusedapureclient-sideSPAsite.This

bookwillfocusontheclient-sideSPAusageandshowhowviewsarepopulated

fromclient-sidefetchesofdatafromtheserverHTTP/RestAPIcreatedinpart

twoofthisbook.

YoucanputatthetopofyourApp.jsfileanimportstatementtopullinthe

needed components that are provided by the react-router-dom package. These

wouldbeHashRouter,RouteandLink.Hereisthecodethatcouldbeplacedinto

App.jsforasimpleusagethathasnoCSSstylingforit:

//App.js

importReact,{Component}from'react';

import'./App.css';

import{HashRouter,Route,Link}from'react-router-dom'

constHome=()=>(

<div>

</div>

)

constAbout=()=>(

<div>

<h2>About</h2>

</div>

)

classAppextendsReact.Component{

render(){

return(

<div>

<Linkto="/"replace>Home</Link>

<Linkto="/about"replace>About</Link>

<hr/>

<Routepath="/about"component={About}/>

<Routeexactpath="/"component={Home}/>

<Routecomponent={NotFound}/>

</div>

</HashRouter>

);

}

exportdefaultApp;

TheHashRouter is the main component wrapper that creates the container

for the UI to be rendered as navigation happens. Different components are

renderedasassignedbythe“component”attributeoftheRoute.Forasitethat

youwantbrowserhistorykept,useBrowserRouterinsteadofHashRouter.Inthe

above case, we are discussing a SPA, which would be implemented with the

hashrouting.

Theattribute“exact”isusedtomatchonlyasinglespecificpath.Ifyouhave

multiple components being rendered at a time, this is needed and will fix that

problem.

TheLinkcomponentisusedasananchortagmightalsobeused,butadds

theprop“to”forspecifyingtheroutetotransitionto.TheLinkcomponentworks

hand-in-hand with the Route component which specifies what component to

render for a given route. The “replace” attribute of the Link specifies that

clicking the link will replace the current entry in the history stack instead of

addinganewone.ThisiswhatwewantforaSPAclient-sidesite.

ThelastRoutelistedisonethatletsallotherpathsberenderedwithanerror.

Ifyouendupthere,youwouldprovideaNotFoundcomponenttostatethatthis

isa“404”,becauseitwasnotexpected.Forexample,thisNotFoundUIwould

renderifyouwenttosomethinglikelocalhost:3000/#/blahblahblah.

HereiswhattheUIendsuplookinglikeforthispreviousexamplecode:

Figure93-ReactRouterusage

16.2UsingBootstrapwithReact

Stylingawebsitetocreateamodernresponsivesitethatlooksgoodinafull-

screenbrowseraswellasonasmallmobilephoneisachallenge.Thereismuch

toconsiderwhenconstructingyourHTMLinthiscase.Thisbookisnotaway

foryoutolearnCSSorresponsiveWebDesigntechniques.Whatthisbookwill

doinsteadistakeasimpleandeffectiveapproachbyusingBootstrap.

Bootstrap was created by Twitter and is a set of UI HTML templates and

CSSstylesthatyoucanusetoaccomplishresponsivewebsitedesigns.

TomakethiseveneasierinaReactapplication,thereisannpmpackagethat

exportsReactComponentstobeeasilyused.Youjustneedtorunthefollowing

installtogeteverythingavailableintheproject:

npminstall--savereact-bootstrapbootstrap@3

TheonlyotherchangetomakeistopullintheCSSfilesneededsothatthey

areexposedforusageintheapplication.Hereisthealteredindex.jsfilethatis

foundinthesrcdirectory.Theaddedlinesareinbold.

//index.js

importReactfrom'react';

importReactDOMfrom'react-dom';

importAppfrom'./App';

import'bootstrap/dist/css/bootstrap.css';

import'bootstrap/dist/css/bootstrap-theme.css';

import'./index.css';

ReactDOM.render(

<App/>,

document.getElementById('root')

);

HereisanalteredApp.jsfilethatnowshowstheusageofaBootstrapstyled

button:

//App.js

importReact,{Component}from'react';

import{Button}from'react-bootstrap';

classAppextendsComponent{

render(){

return(

<div>

<h1>WelcometoReactwithBootstrap</h1>

<p><Button

bsStyle="success"

bsSize="large"

href="http://react-bootstrap.github.io/components.html"

target="_blank">

ViewReactBootstrapDocs

</Button></p>

</div>

);

}

exportdefaultApp;

Hereiswhattherenderingoftheexampleabovewouldlooklike:

Figure94-BootstrapusagewithReact

You can visit this React-Bootstrap site (https://react-

bootstrap.github.io/components/alerts/) to see all of the available components

youcanuseinyourReactapplication.TheNewsWatcherapplicationwillmake

useofseveralofthem.

16.3MakingHTTP/RestRequests

EveryapplicationthatyoucreatewillmostlikelyneedtomakeHTTP/Rest

requeststosomeback-endservicetoretrievedata.Reactdoesnothaveanysuch

capability built in to do that. This was done on purpose, as there are several

librariesthatarealreadygoodatthis.Oneoptionistousethe,builtinstandard

of fetch() as found in the JavaScript standard. Other npm modules such as

superagentcanbeinstalledandused.Thisbookwillutilizefetch().

Typically,youwillneedtofetchdatainthecomponentDidMount()method.

That code will update some internal state properties and in turn would cause

React to do any DOM updates necessary. Here is an example that outputs the

fetcheddatatotheconsolelog:

importsuperagentfrom'superagent';

...Codeleftout...

componentDidMount(){

fetch(`/api/blah`,{

method:'GET',

headers:newHeaders({

'x-auth':this.props.session.token

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

console.log(response.json);

})

.catch(error=>{

console.log(`GETfailed:${error.message}`);

});

}

This type of code will be used in when NewsWatcher is put together. The

basicusageis thatyouspecify theURIor relativepath.Aspart ofthe second

parameterobject,yougivetheHTTPverbusage,suchasput,post,getordelete.

Youcanalsosetanyheadersyouneed.Fetchcanbeusedaspromisebasedcode,

and that is what is shown here. You can verify the HTTP code returned and

handleerrorsandgetatthebodyofthereturneddata.

16.4StatemanagementwithRedux

Youhavelearnedhowtousepropsandstate.Stateandsometimespropsthat

aresharedshouldnotbekeptaspartofacomponentclass,butinsteadshouldbe

keptinacentralstoragelocationforallcomponentstogetat.Ifyoufindthatthe

propsorstateofacomponentreallyneedstobesharedwithothercomponents,

theyshouldbestoredcentrally.

Redux is a library that helps you share state across all your React

components.Itspurposeistoprovideacentralrepositoryforallcomponentsto

writetoandreceiveupdatesfrom.Thinkofitasfulfillingthesamefunctionthat

state does in a component, but across all components in a shared way. With

Redux,youcanalsoaccomplishdatatransformationsandroutingintheprocess.

Thestatethatyoustorecanbesomedatayoucollectedfromuserinteraction,

ordatathatcomesfromanetworkrequestandmuchmore.Justaboutanytype

ofsimpleJavaScriptdataaspartofanobjectcanbekeptthere.

Imagine retrieving data that needs to be shared across the hierarchy of

components.Forexample,theUIinonepartofthecomponenthierarchymight

havestatethatneedstobesharedwithanynumberofothercomponentsstrewn

alloverinthehierarchy.Itwouldgetconfusingifyoutriedtosendthatstateup,

across and down to another component. That would quickly become

incomprehensible.

Reduxsolves these problems by being the central state repository for your

completehierarchyacrossallcomponents.TouseRedux,youfirstneedtocreate

the Redux store. You do this in your code where it is initialized. When you

createthestore,youpassintoitthereducers.Areduceriswhatgetsachanceto

determinethestatechangesyouwanttomakewhendataflowsintheformofan

action. This reducer could be a single function, or a combination of reducers.

Thiswillbecomeclearerasyoulookatsomecodeexamples.

Thenpm packagethatyou installis called “redux”.The Redux objectthat

you can access has only four methods on it, so it is fairly easy to understand.

There are then just a few nuances to understand about each of the four calls.

Here is the creation of a store and also some code showing the four Redux

functions:

import{createStore}from'redux'

conststore=store=createStore(reducer);

store.dispatch(action)

store.subscribe(listener)

store.getState()

store.replaceReducer(nextReducer)

ThecreateStore()callneedsoneormorereducersgiventoit.Itcanalsobe

passed some middleware. There are other npm packages you can download to

actasmiddleware,oryoucanwriteyourown.Thisjustmeansthattheflowcan

beinterceptedandalteredinsomeway.Thisissimilartowhatmiddlewaredoes

inExpresswithNode.js.Ihavenotneededtousethiscapability.

Thedispatch()calliswhatyouusetocauseachangeinstatetohappen.The

getState() function is a way to fetch the state at that moment in time and the

subscribe()isforreceivingasynchronousnotificationsasstateischanged.

In the NewsWatcher codebase, I make the initial call to createStore()with

someprovidedreducerfunctionsandthencalldispatch()alot.Tereisalsousage

ofanothermoduleontopofReduxthatIwillsoondescribe.

Theflowofastatechange

WhatyouwouldtypicallydoindevelopingyourReactcodeistocodeupa

component and use this.state for keeping data that changes local to that

component. You might recall that you do that in a component with the

this.setState()call.Thenasyouprogressandfindthatthestateissomethingthat

needstobeshared,youmoveittotheReduxstatestorage.Don’tputeverything

inReduxasthatcanbecomeoverwhelming.

To update the Redux state, you make a call to store.dispatch(action). For

example,if acomponent wanted toupdate some stateand make thatavailable

backtoitselfandtoothercomponents,itwouldmakethatcall.Thecomponent

calling dispatch() does not need to know about who will listen for that state

change.Thisisasingledirectionofflowwiththestatechange.Therearenobi-

directionalmessagesflowingthroughReduxinanykindofconversation.

The simple usage is to preconfigure Redux with what are called reducers.

Theseactasawaytofilterdataasitflowsthrough.Dispatchcallshappenand

flow through what is called an “action” in the reducers which then causes the

particularaffectedstatetochangeandbeavailable.

Codereceives thechanges inthe state viaa store.subscribe()call.Youcan

alsogeta snapshotofthe stateata giveninstantwith thecallstore.getState().

Here is a visual representation of flow for a state change as it passes via the

Reduxstore:

Figure95-Reactstatemanagement

Here is the simplest code that can be used to demonstrate the basic flow.

Notethattheactionobjectmustatleasthaveapropertynamed“type”.Thisis

usedtofindthecorrectroutertouseforprocessing.Theotherpropertiesofthe

actionobjectareuptoyoutopassanddealwith.Thefollowingcodecreatesthe

Reduxstorewiththepassedinreducer.Thefunctionusagehasaninitialobject

passedinthatsetsthestorageofthestateproperty.

constinitialState={

count:0,

currentMsg:"HelloRedux"

}

conststartState=(state=initialState,action)=>{

switch(action.type){

case'INCREMENT':

return{

...state,

count:state.count+action.incAmount;

}

default:

returnstate;

}

const{createStore}=Redux;

conststore=createStore(startState);

constCounter=({

value,

onIncrement

})=>(

<div>

<h1>Howmanybugshaveyoufixedtoday?</h1>

<h1>{value}</h1>

<buttononClick={onIncrement}>+</button>

</div>

);

constrender=()=>{

ReactDOM.render(

<Counter

value={store.getState().count}

onIncrement={()=>store.dispatch({type:'INCREMENT',incAmount:1})

}

/>,

document.getElementById('root')

)

};

store.subscribe(render);

render();

Thestateforagivenreducercantakeanyformyoudefineforit.Youcansee

inthepreviouscodethatitisdefinedwithdefaultvaluestobeinitializedwith.

Reduxmakesaninitialcalltothereduceronitsownandthedefaultsareset.

Toalterthestate,adispatchcallhappensandpassesintheactionobjectwith

thetypeand valuestoconsume. IntheRedux code,the stateisimmutable,so

youcannotchangeitdirectly,youhavetomakeacloneofit(i.e.seeJavaScipt

Object.assign()usage,suchasObject.assign({},state,{users:action.users});),

or use something like the immutability-helper. This will be used in the

NewsWatcher sample application. You also need to return the complete state

objectback, as it will all be replaced. The code above simply creates the new

object from properties of the existing state using the object spread syntax of

Javascript.Thenthecountisupdated.

Forexample, if itwas an object, you replacethe whole thing, notjust one

property. With the what you saw with React component state, you have one

objectthathad independentlyupdatable propertieson it.Reduxdoes notwork

thesameway.Youcan,however,havedifferentreducers thatareindependent.

Updatingone,doesnotaffecttheother.

MostapplicationswillwanttosplitoutwhatisstoredintheReduxstorage

into separate concerns. You simply do this by calling combineReducers() to

piecethemalltogether.Eachonewillreceivealloftheactionsroutedtothem,

butitisofcourseuptothereducerstomatchwhataction.typevaluestheycare

about.YoucallcreateStorewiththiscombinedrootReducerinsteadofasingle

reducerfunction.

import{combineReducers}from'redux'

constrootReducer=combineReducers({

app,

news,

sharednews,

profile

})

UsageofReduxwithreact-redux

The Redux library is a reusable JavaScript library that can be used in

JavaScript code. It is certainly usable as it is with is four methods. There are,

however,certaincodedesignpatternsofReactthatareusedoverandover.One

ofthesepatternsinvolvesstatemanipulations.Inordertocombinethatpattern

withtheuseofRedux,inaneasytoconsumeway,therewasalibrarycreated

namedreact-redux.

Togetstarted,youneedtoinstallthenpmpackagesreduxandreact-redux.

You then need to add some code to create the single Redux store. There is a

specialReactcomponentthatisusedtowrapyourapplicationcomponentsoas

to make Redux available in all of the component hierarchy. Here is what the

index.jsfileendsuplookinglike.Thekeychangeshavebeenhighlightedinbold

font.

importReactfrom'react';

importReactDOMfrom'react-dom';

import{createStore}from'redux'

import{Provider}from'react-redux'

importreducerfrom'./reducers'

importAppfrom'./App';

import'bootstrap/dist/css/bootstrap.css';

import'bootstrap/dist/css/bootstrap-theme.css';

import'./index.css';

conststore=createStore(reducer);

ReactDOM.render(

<Providerstore={store}>

<App/>

</Provider>,document.getElementById('root')

);

The App component will be a child of the Provider component. The code

fromreact-reduxlibrarywillbeabletouseitandpassinadditionalprops.Itwill

passinapropthatisafunctionnamed“dispatch”.Thisway,youdon’tneedto

pass the Redux store to the App component. Youwon’tactually be coding up

callstostore.dispatch(),justdispatch().

Thenextthingyouneedtounderstandishowtotakeacomponentyouwrite

anddotheworktogetthestatefromthechangesthathappenintheReduxstore

andtransferthatstatetothelocalpropsonthecomponent.

The state coming from the Redux store can be used in the props on a

component. Props are then changed via the dispatch to cause the behavior to

change. You do not take Redux state and transfer it to the this.state of a

component. That would be a waste, as they both serve the same purpose and

wouldbeduplicatedandyouwouldnothaveasinglesourceoftruth.

Hereiswhatyouneedtodoinacomponenttohaveitpullinstatechanges

fromaReduxstore:

import{connect}from'react-redux'

...lotsofcomponentcodeleftout...

MyComponent.propTypes={

dispatch:PropTypes.func.isRequired

};

constmapStateToProps=state=>{

return{

session:state.app.session,

user:state.profile.user,

isLoading:state.profile.isLoading

}

exportdefaultconnect(mapStateToProps)(MyComponent)

Thiscodemayseemalittlestrangewhenyoufirstlookatit.Whatyouare

doing here is instead of a simple export of MyComponent, you call the react-

reduxprovidedconnect()functionandthathastheabilitytobeabletotakeyour

componentandwrapitforitsownpurposes.Reduxkeepstrackofwhenitneeds

tosendstateupdatesandre-renderitandprovidethedispatchprop.Youcansee

thatyoucangetwhateveryoulikefromtheReduxstore.Thistimeitispulling

fromtheappandprofilesettings.

Youpasstotheconnect()callacallbackfunctionthatletsyougetaccessto

theReduxstate.ThisfunctioniscalledeverytimeanythingintheReduxstate

changes. Using the state passed in, you can take whatever your particular

componentcaresabout.Thiscontainsallofthedifferentstoredstatefromallof

thereducersthatareinuse.

Thereismorethatispossible withreact-redux.Forexample,youcan pass

other parameters to connect(), such as a function that allows you to hook up

dispatch calls from your component, such as UI click handlers that make

dispatchcalls.Thensomewhereinyourcode,youcancallthepreparedfunction

thatisavailableontheprops.

import{connect}from'react-redux'

...lotsofcomponentcodeleftout...

sendAction=()=>{

this.props.sendAction()

}

render(){

<div>

<ButtononClick={sendAction}/>

</div>

}

MyComponent.propTypes={

dispatch:PropTypes.func.isRequired

};

constmapStateToProps=state=>{

return{

session:state.app.session,

user:state.profile.user,

isLoading:state.profile.isLoading

}

functionmapDispatchToProps(dispatch){

return({

sendAction:()=>{dispatch(

{type:'SOME_ACTION',someData:1})}

})

}

exportdefaultconnect(mapStateToProps,

mapDispatchToProps)(MyComponent)

Youdon’thavetosetthingsupthisway,asdispatch()isavailableanywayas

a prop of the component already. Here is an example that adds in the

mapDispatchToPropsfunction:

Ifyoudon’twanttosetupamapDispatchToPropsthenyoujustcalldispatch

fromthis.propsasfollows.

sendAction=()=>{

this.props.dispatch({type:'SOME_ACTION',someData:1})

}

Note: The usage of additional libraries like Redux is completely at your

discretion.Onlyusealibraryifatanypointyouseethatyoucansimplifyyour

code. Sometimes a simple code base with no design frills works just fine. You

must decide when and how to refactor and introduce new libraries into your

code and what the cost versus benefit will be. In the case of Redux, I believe

thereisaclearbenefit.YouwillseeitusedthroughouttheNewsWatchercode.

Chapter17:NewsWatcherApp

DevelopmentwithReact

With the previous chapter’s content, you are now able to understand the

constructionoftheNewsWatcherpresentationlayercode.Ifyoufollowedalong

inthepreviouschapterinthesectiononinstallationsteps,youareallsettobegin

writingcode.

In this chapter, you will learn about the files as they exist in the project

postedonGithubandhoweachoneispiecedtogether.Ifyouclonetheproject

fromGitHub,theprojectfolderswilllookasfollows:

Figure96-VSCodefiletree

Note: Don’t forget that you can access all of the code for the sample

NewsWatcherprojectathttps://github.com/eljamaki01/NewsWatcher2RWeb.

Youmay havenoticedthat youhad aplaceholder index.htmlfilethat your

Node.jsservice was serving up. The existing Node.js project frompart two of

this book is capable of serving up your UI after you make just a few minor

tweaks.

17.1WhereitAllStarts(src/index.js)

BackinthesectionsontheNode.jswebservicedevelopment,yousawafew

linesofcodeintheserver.jsfilethatservedupagetrequestforthemainHTML

page.ThisallowedforthedownloadinganddisplayoftheNewsWatchersite.

Theexpressroutingforthatfilewasseparatedfromthoseprovidingaccess

tothebackendAPI.Herearethelinesfromserver.jsfortheroutehandlingthat

servesuptheindex.htmlfileanditsassociatedstaticresources:

app.get('/',function(req,res){

res.sendFile(path.join(__dirname,'build','index.html'));

});

app.use(express.static(path.join(__dirname,'build')));

Theapp.get()callgivesaspecificrouteforagetrequesttoyourrootsiteand

statesthatyouwillalwaysserveupyourindex.htmlfileforthat.Thisisreally

allyouneedtosetuptheuseofReacttogetthereactapplicationsenttoberun

onaclient-sidebrowser.

Theapp.use()call sets upmiddleware for arouter path forall of thestatic

files. You simply tell it that you have this directory named “build” for where

theyallarelocated.

Youwillnotethatthisfinalindex.hmlfileisnotafileyoucreated,butitis

generatedfor youin the buildprocess fromthe index.js filethat youprovided

and the template index.html file found in the public folder. Look in the build

foldertofindthegeneratedindex.htmlfilethatgetsservedup.

Hereistheindex.jsfile.Thefewadditionsyouwouldfindherearejustthe

inclusion of css files that can be used across anything that is rendered. In

particular,youseetheonesforstylingbootstrapelements.Youalsoseethecode

tosetupReduxtobeusedacrossalltheapplication.Thereducersarebroughtin

andtheProvidercomponentissomethingfromreact-reduxthatprovidesRedux

totheoverallapplication.

importReactfrom'react';

importReactDOMfrom'react-dom';

import{createStore}from'redux'

import{Provider}from'react-redux'

importreducerfrom'./reducers'

importAppfrom'./App';

import'bootstrap/dist/css/bootstrap.css';

import'bootstrap/dist/css/bootstrap-theme.css';

import'./index.css';

conststore=createStore(reducer);

ReactDOM.render(

<Providerstore={store}>

<App/>

</Provider>,document.getElementById('root')

);

ThecalltoReactDOM.render()isthestartingpointcalltoreactthatrenders

totherootdivelement.AllofyourUIemanatesfromhere,andcomesfromthe

Appcomponent.

17.2Thehubofeverything(src/App.js)

The App.js file is where we establish the UI that renders the views. This

meansthatitprovidesthingslikethecapabilitytopresentanavigationbarand

allowsforthenavigatingbetweenthedifferentpageviews,suchasloggingin,

viewingnews,andsettingprofilesettings.ThisfilesetsuptheAppclassthatis

derivedfromtheComponentclass.

Imports,constructorandthecomponentDidMountlifecycle

Theimportsaresetuptobringinexternallibrariesandtheothercomponents

thatareneededforviewsthatareusedinthenavigation.Thenthereisthecode

fortheAppcomponentclass.

Reduxisusedforthestatesettingsthatareneeded.Youwillfindtheuseof

thedispatch()functiontostorestatedatainReduxinacentralwaythatisglobal

totheapplication.LookatthemapStateToProps()functionatthebottomandyou

willseewhatpropertiestheAppclassretrievesfromRedux.Theyare:

Thestateobjecttotellusifwearesignedin.

Thesessiontokenthatwasretrievedfromtheserver-sidelogin.

ThestatusmessagethatisdisplayedatthetopoftheUI.

The componentDidMount() lifecycle event method is where you do

processingoncetheUIhasbeenrenderedandneedtomakeanyalterations.The

codechecksthelocalbrowserstoragetoseeifatokenhasbeensavedawayto

beretrievedandifsogetsthatandsetsamessagethattheuserisloggedin.The

userwouldneedtohaveselectedtheoptionatlogintimetohaveonesaved.The

dispatchthathappensplacesthattokenintoReduxstoragetoglobalaccessfrom

allotherpagesthatneedit,suchasforprofileretrieval.

The logged in flag is set to true in that same dispatch processing, which

altersthemenurendering,asyouwillsoonsee.ThevalueofcurrentMsgisset

throughthedispatchandwouldshowupintheUI.

TheUIisrenderedoncefromtheserver,withtheinitialHTML,withitsCSS

andJavaScriptandthenitiscompletelyself-sufficientintheclientbrowserfrom

thenon.Therearenopagesbeingrenderedfromtheserversideafterthat.You

could create a combination of server and client-side page rendering in a true

isomorphicapplicationaswillbediscussedlater.Hereisthecodethathasbeen

discussedthusfar.

//App.js

importReact,{Component}from'react';

import'./App.css';

import{HashRouter,Switch,Route}from'react-router-dom'

import{Navbar,Nav,NavItem}from'react-bootstrap';

import{IndexLinkContainer}from'react-router-bootstrap';

import{connect}from'react-redux'

importLoginViewfrom'./views/loginview';

importNewsViewfrom'./views/newsview';

importHomeNewsViewfrom'./views/homenewsview';

importSharedNewsViewfrom'./views/sharednewsview';

importProfileViewfrom'./views/profileview';

importNotFoundfrom'./views/notfound';

classAppextendsComponent{

componentDidMount(){

//CheckfortokeninHTML5clientsidelocalstorage

conststoredToken=window.localStorage.getItem("userToken");

if(storedToken){

consttokenObject=JSON.parse(storedToken);

this.props.dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as

${tokenObject.displayName}`,session:tokenObject});

}else{

}

...codeleftout...

}

constmapStateToProps=state=>{

return{

loggedIn:state.app.loggedIn,

session:state.app.session,

currentMsg:state.app.currentMsg

}

exportdefaultconnect(mapStateToProps)(App)

This application is a SPA application, so hash based browser navigation is

used. This does not allow for history to be kept and if you try and use the

browserbackbutton,theapplicationwouldnothaveanyhistorytogobackand

forthfrom.

Navigationbar

Themain point of this App component is to set up the navigation bar that

will exist for users to get access to the menus and see a status message. The

DOMrenderingmakesuseofsomehandybootstrapstyling.Youcaninvestigate

theparticularsofhowitallworksonyourownthroughtheusageofthereact-

bootstrap npm package. Here is an image of what would be rendered on a

smartphonedevice.ItshowstheUIstatewhenthemenuisopened.

Figure97-Sampleimagefromasmartphone

The react-boostrap usage has the ability to do what is called “responsive”

webrenderinganditcreatesabuttonthatwillbeusedtoprovideadrop-down

menuwhen it isbeing run ona mobile smartphone.If you runit in adesktop

browser, it does not appear this way, but the navigation bar is spread out.

Bootstrapclassesadapttothesizeofthedisplay.

Note:Thegreatthingisthatyoudon’tneedtousetherawbootstrapstylings

withlow-levelHTMLelements.React-boostrapwrapsallofthatandpresentsit

foryourusageinreactComponentsthatareconsumed.

Aspartoftheheaderofthenavbar,thereisaspanelementusedtodisplay

messagesyouwanttheusertosee.Forexample,iftheirloginfails,youwantthe

usertoknowthat.ThestateissetthroughReduxwiththeappropriatemessage

string.

EachIndexLinkContainerentryrepresentsthemenuselectionsfornavigating

around the application. There is a clever way to show or hide each entry

dependingonthestateoftheapp.TheloggedInpropertyisusedtodeterminethe

displayofnavigationbarentries.Whenauserisloggedin,youwantallofthe

menuselectionsvisible.Untilthen,theyarehidden.

The actual navigation click-handling is done through the use of the react-

router-domnpmpackage.Attheverytopofwhatisrendered,istheusageofthe

HashRoutercomponent.Thisiswhatwrapseverythingandgivesustheability

torender in the DOM whatever weset up as a page view route.Thenwe can

either let the navigation bar selections control the route to select, or in some

cases,managethenavigationprogrammatically.

You can see the react-router-dom Switch component that is used in

conjunctionwiththeRoutecomponenttohandlewhatpageviewsgetrendered.

TheinterestingthingisthatforeachRoute,youspecifythepaththatitisforand

the component to render. In one case, the code has to actually use the render

Propinordertopassthestateintotheusageofthepageviewcomponent.Here

istherender()functionfortheAppcomponentthatcontrolsthenavigation:

render(){

return(

<div>

<Navbar.Header>

<Navbar.Brand>

NewsWatcher{this.props.currentMsg&&<span><small>({this.props.currentMsg})</small>

</span>}

</Navbar.Brand>

<Navbar.Toggle/>

</Navbar.Header>

<Navbar.Collapse>

<Nav>

{<IndexLinkContainer to="/" replace><NavItem >Home Page News</NavItem>

</IndexLinkContainer>}

{this.props.loggedIn&&

<IndexLinkContainerto="/news"replace>

<NavItem>MyNews</NavItem>

</IndexLinkContainer>}

{this.props.loggedIn&&

<IndexLinkContainerto="/sharednews"replace>

<NavItem>SharedNews</NavItem>

</IndexLinkContainer>}

{this.props.loggedIn&&

<IndexLinkContainerto="/profile"replace>

<NavItem>Profile</NavItem>

</IndexLinkContainer>}

{this.props.loggedIn&&

<NavItemonClick={this.handleLogout}>Logout</NavItem>}

{!this.props.loggedIn&&

<IndexLinkContainerto="/login"replace>

<NavItem>Login</NavItem>

</IndexLinkContainer>}

</Nav>

</Navbar.Collapse>

</Navbar>

<hr/>

<Routeexactpath="/"component={HomeNewsView}/>

<Routepath="/login"component={LoginView}/>

<Routepath="/news"component={NewsView}/>

<Routepath="/sharednews"component={SharedNewsView}/>

<Route path="/profile" render={props => <ProfileView appLogoutCB={this.handleLogout}

{...props}/>}/>

<Routecomponent={NotFound}/>

</Switch>

</div>

</HashRouter>

);

}

Note: Look at the top of the top of the file App.js file to see the import

statements. These will help you keep straight which components come from

which npm packages. For example, those from react-bootstrapand those from

react-router-dom.

Loggingout

Thenavigationbarhasaselectiontologtheuserout.Thissimplyplacesa

calltothebackendserviceanditcandowhateveritwouldlikecodewise,but

theclientsidereallyjustneedstosetthestatetoreflectthatandredirecttheview

to the login. This is done through a browser capability with the use of setting

window.location.hash.

handleLogout=(event)=>{

const{dispatch}=this.props

event&&event.preventDefault();

fetch(`/api/sessions/${this.props.session.userId}`,{

method:'DELETE',

headers:newHeaders({

'x-auth':this.props.session.token

}),

cache:'default'//no-storeorno-cache?

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

dispatch({type:'DELETE_TOKEN_SUCCESS',msg:"Signedout"});

window.localStorage.removeItem("userToken");

window.location.hash="";

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',msg:`Signoutfailed:${error.message}`});

});

}

17.3ReduxReducers(src/reducers/index.js)

You saw in the src/index.js file how the reducers were brought into the

application when it started up. The line was as follows that set up the Redux

Store:

importreducerfrom'./reducers'

conststore=createStore(reducer);

Incaseyouforgot,ifyouhavearequireorimportstatementofadirectory,it

willbydefaultlookforafilenamedindex.jsandusethat.

Index.jsthencombinesalltheseparatereducersintooneandexportsthat:

//src/reducers/index.js

import{combineReducers}from'redux'

importappfrom'./app'

importhomenewsfrom'./homenews'

importnewsfrom'./news'

importsharednewsfrom'./sharednews'

importprofilefrom'./profile'

constrootReducer=combineReducers({

app,

homenews,

news,

sharednews,

profile

})

exportdefaultrootReducer

Let’snowlookatafewofthereducersbringcombined.

TheAppReducer

The App reducer has three actions that it supports – MSG_DISPLAY,

RECEIVE_TOKEN_SUCCESS and DELETE_TOKEN_SUCCESS. We have

seen these used in the src/app.js file where the redux state is used for each of

these.InthecaseoftheMSG_DISPLAY,itsimplysetsanewstringforthetext

ofthecurrentMsgproperty.

Notice the “…state” code that is a JavaScript way of taking an object and

pullinginallofitsproperties.Thisway,youdon’thavetolistthem.Thisway

you get whatever was set for session and loggedIn and then currentMsg is

overridden.

//src/reducers/app.js

constinitialState={

loggedIn:false,

session:null,

currentMsg:""

}

constappLevel=(state=initialState,action)=>{

switch(action.type){

case'MSG_DISPLAY':

return{

...state,

currentMsg:action.msg

}

case'RECEIVE_TOKEN_SUCCESS':

return{

...state,

loggedIn:true,

session:action.session,

currentMsg:action.msg

}

case'DELETE_TOKEN_SUCCESS':

return{

...state,

loggedIn:false,

session:null,

currentMsg:action.msg

}

default:

returnstate

}

exportdefaultappLevel

Iwillshowonemorereducer.Thisonewillillustratethepointthatthestate

in the Redux store is immutable, so you have to either replace the complete

object,oruseaspecialmoduletohelpyoumanagethat.

Thereisannpmmodulenamedimmutability-helperthatallowsyoutomake

achangetoanimmutableobject.Partofthisstatesettingisanewspropertythat

isanarrayofnewsstorieswiththeircomments.

AnewcommentisaddedwiththeADD_COMMENT_SUCCESSaction.It

doesthatbypushingittotheendofthearrayforanewsstory.Inotherwords,

thereisanarrayofnewsstoriesandeachofthosehaveanarrayofcomments.If

we did not use the immutability helper, a completely new copy of the array

wouldhavetobemadeandreplacedeachtime.

//src/reducers/sharednews.js

importupdatefrom'immutability-helper';

constinitialState={

isLoading:true,

news:null

}

constnews=(state=initialState,action)=>{

switch(action.type){

case'REQUEST_SHAREDNEWS':

return{

isLoading:true,

news:[]

}

case'RECEIVE_SHAREDNEWS_SUCCESS':

return{

//...state,

isLoading:false,

news:action.news,

}

case'ADD_COMMENT_SUCCESS':

return{

...state,

news:update(state.news,{

[action.storyIdx]:{

comments:{

$push:[{displayName:action.displayName,comment:action.comment}]

}

})

}

default:

returnstate

}

exportdefaultnews

Theimmutabilityhelpersyntaxisabittrickyhere.Whatisbeingalteredis

an array property found in an object in the new array. The first thing to do is

selecttheindexofthestoryinthenewarray,andthentopushacommenttothe

commentsarrayfoundinthatobjectentry.

Alessefficientwaywouldbetoclonethearrayandthenalterit.Somearray

functionslikeslice()actually returnanewarrayand don’tmutatetheold one.

Youcanthusgetthenewarrayandthenalteritasyoulike.

Youcan use the JavaScipt Object.assign() that allows you to create a new

objectfromtheexistingstateobjectandthenoverridethecommentspropertyof

that.Hereishowthecodewouldbeforanarrayusingtheslice()function:

case'ADD_COMMENT_SUCCESS':

varnewNews=state.news.slice(0);

newNews[action.storyIdx].comments.push({ displayName: action.displayName, comment:

action.comment});

return{

...state,

news:newNews

}

Therestofthesectionsinthischaptergooverthecodethatexistsineachof

theviewsthatarerendered–login,homeNews,news,sharednewsandprofile.

Inthe previous chapterson React fundamentals,you learned about Reduxand

refactoringyourcomponenttosplitthemintocontrollerandviewcomponents.I

have chosen not to complicate the code by splitting out each view into its

controller/containercomponent.Ikeepallthecodeinonesinglecomponent.

17.4TheLoginPage(src/views/loginview.js)

To log a user in, their email and password are entered and verified by the

backendservice.Theloginpagecontainsacheckboxfortheusertospecifythat

theywanttheirlocaldevicetostoretheirlogintokenforthem.Iftheuserisnot

registeredyet,theycanclicktobringupapopupmodaldialogformtoregistera

newaccount.Aspartoftheregistration,anemailneedstobeprovided.Emails

areuniqueinthesystem,sonoduplicatesareallowedinthedatalayer.Hereis

whattheUIlookslikeforloggingin.

Figure98-NewsWatcherloginform

The HTML code for the form is set up to have a submit handler. The

handleLogin()functionofthe componentis thecode thatexecutesat thattime.

To get the data from the form, various Bootstrap components are used. These

bindtheirdatatothestateofthecomponentthroughonChangehandlersthatget

setaseachcharacteristyped.Hereistherendercodefortheloginpageasfound

intheloginview.jsfile:

render(){

//Ifalreadyloggedin,don'tgohereandgetroutedtothenewsview

if(this.props.session){

returnnull;

}

return(

<div>

<formonSubmit={this.handleLogin}>

<FieldGroup

id="formControlsEmail2"

type="email"

glyph="user"

label="EmailAddress"

placeholder="Enteremail"

onChange={this.handleEmailChange}

<FieldGroup

id="formControlsPassword2"

glyph="eye-open"

label="Password"

type="password"

onChange={this.handlePasswordChange}

<Checkboxchecked={this.state.remeberMe}

onChange={this.handleCheckboxChange}>

Keepmeloggedin

</Checkbox>

<ButtonbsStyle="success"bsSize="lg"blocktype="submit">

</Button>

</form>

<p>NotaNewsWatcheruser?

<astyle={{cursor:'pointer'}}

onClick={this.handleOpenRegModal}>SignUp</a>

</p>

{this._renderRegisterModal()}

</div>

);

}

You can see that at the top of the render() function we first see if we are

loggedin.Whenweareloggedin,thesessionpropertyfromReduxisset.

Theotherveryinterestingthingthatisgoingonisthatyouseethecalltothe

function renderRegisterModal(). I could have just placed all the HTML right

there in the render function. I did not do that as that function is large enough

already.Breakingit out makesthe codeeasier toread. Thistakes that outand

makesitself-contained.

TheotherthingtoknowisthatthisUIforregisteringauserofNewsWatcher

is a modal dialog, and uses the <Modal> component that is provided by

BootstrapforReact.Itisalwaysrendered,butitisbeingshownorhiddenbya

statepropertyandyoucanseethatitstartsoutasbeinghidden.

ThemodalregistrationdialogUIlooksasfollows:

Figure99-NewsWatcherRegistrationModal

HereisthecodefortheregistrationmodalUI:

_renderRegisterModal=()=>{

return(<Modalshow={this.state.showModal}onHide={this.handleCloseRegModal}>

<Modal.HeadercloseButton>

<Modal.Title>Register</Modal.Title>

</Modal.Header>

<Modal.Body>

<formonSubmit={this.handleRegister}>

<FieldGroup

id="formControlsName"

type="text"

glyph="user"

label="DisplayName"

placeholder="Enterdisplayname"

onChange={this.handleNameChange}

<FieldGroup

id="formControlsEmail"

type="email"

glyph="user"

label="EmailAddress"

placeholder="Enteremail"

onChange={this.handleEmailChange}

<FieldGroup

id="formControlsPassword"

glyph="eye-open"

label="Password"

type="password"

onChange={this.handlePasswordChange}

<ButtonbsStyle="success"bsSize="lg"blocktype="submit">

<Glyphiconglyph="off"/>Register

</Button>

</form>

</Modal.Body>

<Modal.Footer>

<Button bsStyle="danger" bsSize="default" onClick={this.handleCloseRegModal}><Glyphicon

glyph="remove"/>Cancel</Button>

</Modal.Footer>

</Modal>)

}

ThisUIuses aformand hasa functionthathandles thesubmit.Let’snow

lookatthecodethatsupportstheLoginViewcomponent.

ComponentsupportingcodeofLoginView

Thereistheusualconstructorthatsetsupwhatisneededforthestate.Inthis

case, we have properties that will hold the bound data from the form, such as

email and password. The showModal property is used to show and hide the

modalregistrationform.

As explained, at the top of the render function is a test to see if there is a

session token and a null is returned, which React sees and ignores. The login

menu item is not shown anyway if the user is already logged in. The session

tokencomesthroughtheusageofRedux.

Oncetheusertypesintheirusernameandpasswordandthenclick“Login”,

thehandleLogin()methodmakesthecalltothebackendtogetatoken.Itsetsthe

email and password in its HTTP POST request. If successful, the message

RECEIVE_TOKEN_SUCCESS is sent through Redux with a dispatch() call.

Thissetsthesessiontokenfortherestoftheapplicationtosee.Then,thereisa

changemadetothebrowserlocationtogotothenewspageview.

ThehandleRegistration()method makes the callto the backend to create a

useraccountinthedatalayer.Theothermethodsyouseeareforthebindingof

thedatatothestateandtheopeningandclosingofthemodalregistrationdialog

viathestatepropertyforthat.

Here is the rest of the code, minus the render code that you have already

seen:

importReact,{Component}from'react';

importPropTypesfrom'prop-types';

import{Checkbox,Button,Modal,Glyphicon}from'react-bootstrap';

import{connect}from'react-redux'

importsuperagentfrom'superagent';

importnoCachefrom'superagent-no-cache';

import{FieldGroup}from'../utils/utils';

import'../App.css';

classLoginViewextendsComponent{

constructor(props){

super(props);

this.state={

name:"",

email:"",

password:"",

remeberMe:false,

showModal:false

};

}

handleRegister=(event)=>{

const{dispatch}=this.props

event.preventDefault();

returnfetch('/api/users',{

method:'POST',

headers:newHeaders({

'Content-Type':'application/json'

}),

cache:'default',//no-storeorno-cacherodefault?

body:JSON.stringify({

displayName:this.state.name,

email:this.state.email,

password:this.state.password

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==201){

thrownewError(response.json.message);

}

dispatch({type:'MSG_DISPLAY',msg:"Registered"});

this.setState({showModal:false});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',msg:`Registrationfailure:${error.message}`});

});

}

handleLogin=(event)=>{

const{dispatch}=this.props

event.preventDefault();

returnfetch('/api/sessions',{

method:'POST',

headers:newHeaders({

'Content-Type':'application/json'

}),

cache:'default',//no-storeorno-cacherodefault?

body:JSON.stringify({

email:this.state.email,

password:this.state.password

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==201){

thrownewError(response.json.message);

}

//Setthetokeninclientsidestorageiftheuserdesires

if(this.state.remeberMe){

varxfer={

token:response.json.token,

displayName:response.json.displayName,

userId:response.json.userId

};

window.localStorage.setItem("userToken",JSON.stringify(xfer));

}else{

window.localStorage.removeItem("userToken");

}

dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as

${response.json.displayName}`,session:response.json});

window.location.hash="#news";

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',msg:`Signinfailed:${error.message}`});

});

}

handleNameChange=(event)=>{

this.setState({name:event.target.value});

}

handleEmailChange=(event)=>{

this.setState({email:event.target.value});

}

handlePasswordChange=(event)=>{

this.setState({password:event.target.value});

}

handleCheckboxChange=(event)=>{

this.setState({remeberMe:event.target.checked});

}

handleOpenRegModal=(event)=>{

this.setState({showModal:true});

}

handleCloseRegModal=(event)=>{

this.setState({showModal:false});

}

...render()and_renderRegisterModal()leftout...

LoginView.propTypes={

dispatch:PropTypes.func.isRequired,

session:PropTypes.object

};

constmapStateToProps=state=>{

return{session:state.app.session}

}

exportdefaultconnect(mapStateToProps)(LoginView)

17.5DisplayingtheNews(src/views/newsview.jsand

src/views/homenewsview.js)

ThisNewsViewcomponentdisplaysthefilterednewspage.TheUIlooksas

follows:

Figure100-NewsWatchernewspage

There is a dropdown list that displays the list of news filters to select

between.TheJavaScriptmap()functionisusedtogothroughthearrayoffilters

and populate that dropdown. Each one is given text from the filter.name

property.ThereisanotheruseofthenewsFilterarraywiththemap()functionto

rendereachofthenewsstoriesfortheselectedfilter.Alinkisprovidedforeach

story,alongwiththeimageandURLtoclickandopen.Thereisalotofusageof

Bootstrapcomponentsinthiscode.Hereistherendermethod:

render(){

if(this.props.isLoading){

return(

<h1>Loadingnews...</h1>

);

}

return(

<div>

<FormGroupcontrolId="formControlsSelect">

<FormControlbsSize="lg"componentClass="select"

placeholder="select"

onChange={this.handleChangeFilter}

value={this.state.selectedIdx}>

{this.props.newsFilters.map((filter,idx)=>

<optionkey={idx}value={idx}>

<strong>{filter.name}</strong>

</option>

)}

</FormControl>

</FormGroup>

<hr/>

<Media.List>

{this.props.newsFilters[this.state.selectedIdx].newsStories.map((story,idx)=>

<Media.ListItemkey={idx}>

<Media.Left>

<ahref={story.link}target="_blank">

<imgalt=""className="media-object"src={story.imageUrl}/>

</a>

</Media.Left>

<Media.Body>

<Media.Heading><b>{story.title}</b></Media.Heading>

<p>{story.contentSnippet}</p>

{story.source}<span>{story.hours}</span>

<Media.Body>

<a style={{ cursor: 'pointer' }} onClick={(event) => this.handleShareStory(idx,

event)}>Share</a>

</Media.Body>

</Media.ListItem>

)}

<Media.ListItemkey={999}>

<Media.Left>

<ahref="http://developer.nytimes.com"target="_blank"

rel="noopenernoreferrer">

<imgalt=""src="poweredby_nytimes_30b.png"/>

</a>

</Media.Left>

<Media.Body>

<Media.Heading><b>DataprovidedbyTheNewYorkTimes</b></Media.Heading>

</Media.Body>

</Media.ListItem>

</Media.List>

</div>

);

}

ThereisalsotheabilitytoclickonalinktoSharestories.

Componentsupportingcode

Whenthecomponentisopened,codeisruntodothefetchingofthenews

stories.ThisisdoneinthecomponentDidMount()method.Youcanalsoseeat

thetop ofthe codehow theconstructor setsup thestate forthe selectednews

filter.TheisLoadingpropertycomesfromReduxandisusedforaUIindication

thatthenewsstoriesarebeingfetched.Themessagegoesawayoncethedatais

available.ThehandleShareStory()methodisprovidedsothatastorycanbesent

tothebackendtobeputinacommonlocationforalluserstoseeandcomment

on.

ThetoHours() is this little helper function that is being used to format the

text to display how old a news story is. Here is the code for the newsview.js,

withtherendermethodleftout,asthatwasalreadyshown:

importReact,{Component}from'react';

importPropTypesfrom'prop-types';

import{FormGroup,FormControl,Media}from'react-bootstrap';

import{connect}from'react-redux'

importsuperagentfrom'superagent';

importnoCachefrom'superagent-no-cache';

import{toHours}from'../utils/utils';

import'../App.css';

classNewsViewextendsComponent{

constructor(props){

super(props);

this.state={

selectedIdx:0

};

}

componentDidMount(){

if(!this.props.session){

returnwindow.location.hash="";

}

const{dispatch}=this.props

dispatch({type:'REQUEST_NEWS'});

fetch(`/api/users/${this.props.session.userId}`,{

method:'GET',

headers:newHeaders({

'x-auth':this.props.session.token

}),

cache:'default'//no-storeorno-cache?

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

for(vari=0;i<response.json.newsFilters.length;i++){

for(varj=0;j<

response.json.newsFilters[i].newsStories.length;j++)

{

response.json.newsFilters[i].newsStories[j].hours =

toHours(response.json.newsFilters[i].newsStories[j].date);

}

dispatch({type:'RECEIVE_NEWS_SUCCESS',newsFilters:response.json.newsFilters});

dispatch({type:'MSG_DISPLAY',msg:"Newsfetched"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',msg:`Newsfetchfailed:${error.message}`});

});

}

handleChangeFilter=(event)=>{

this.setState({selectedIdx:parseInt(event.target.value,10)});

}

handleShareStory=(index,event)=>{

const{dispatch}=this.props

event.preventDefault();

fetch('/api/sharednews',{

method:'POST',

headers:newHeaders({

'x-auth':this.props.session.token,

'Content-Type':'application/json'

}),

cache:'default',//no-storeorno-cacherodefault?

body:JSON.stringify(

this.props.newsFilters[this.state.selectedIdx].newsStories[index])

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==201){

thrownewError(response.json.message);

}

dispatch({type:'MSG_DISPLAY',msg:"Storyshared"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`Shareofstoryfailed:${error.message}`});

});

}

render(){

...TAKENOUT...ALREADYSHOWN...

}

NewsView.propTypes={

dispatch:PropTypes.func.isRequired

};

constmapStateToProps=state=>{

return{

session:state.app.session,

newsFilters:state.news.newsFilters,

isLoading:state.news.isLoading

}

exportdefaultconnect(mapStateToProps)(NewsView)

Thehomenewsstorypageisjustastrippeddownversionofthecodeseen

above, so there is no need to explain it. That is found in the HomeNewsView

component.

17.6SharedNewsPage(src/views/sharednewsview.js)

Thesharednewsstoryviewhasthesametypeofnewslistingcapabilityyou

haveseenbefore.Hereistherendercode:

render(){

if(this.props.isLoading){

return(<h1>Loadingsharednews...</h1>);

}

return(

<div>

<h1>SharedNews</h1>

<Media.List>

{this.props.news.map((sharedStory,idx)=>

<Media.ListItemkey={idx}>

<Media.Left>

<ahref={sharedStory.story.link}target="_blank">

<imgalt=""className="media-object"

src={sharedStory.story.imageUrl}/>

</a>

</Media.Left>

<Media.Body>

<Media.Heading><b>{sharedStory.story.title}</b></Media.Heading>

<p>{sharedStory.story.contentSnippet}</p>

{sharedStory.story.source}–

<span>{sharedStory.story.hours}</span>

<astyle={{cursor:'pointer'}}onClick={(event)=>

this.handleOpenModal(idx,event)}>Comments</a>

</Media.Body>

</Media.ListItem>

)}

<Media.ListItemkey={999}>

<Media.Left>

<ahref=http://developer.nytimes.com

target="_blank"rel="noopenernoreferrer">

<imgalt=""src="poweredby_nytimes_30b.png"/>

</a>

</Media.Left>

<Media.Body>

<Media.Heading><b>DataprovidedbyTheNewYorkTimes</b></Media.Heading>

</Media.Body>

</Media.ListItem>

</Media.List>

{this.props.news.length>0&&

<Modalshow={this.state.showModal}onHide={this.handleCloseModal}>

<Modal.HeadercloseButton>

<Modal.Title>AddComment</Modal.Title>

</Modal.Header>

<Modal.Body>

<formonSubmit={this.handleAddComment}>

<FormGroupcontrolId="commentList">

Comments</ControlLabel>

<ulstyle={{height:'10em',overflow:'auto',

'overflow-x':'hidden'}}>

{this.props.news[this.state.selectedStoryIdx].

comments.map(comment=>

<li>

<div>

<p>'{comment.comment}'-{comment.displayName}

</p>

</div>

</li>

)}

</ul>

</FormGroup>

{this.props.news[this.state.selectedStoryIdx].

comments.length<30&&

<div>

<FieldGroup

id="formControlsComment"

type="text"

glyph="user"

label="Comment"

placeholder="Enteryourcomment"

onChange={this.handleCommentChange}

<Buttondisabled={this.state.comment.length===0}

bsStyle="success"bsSize="lg"

blocktype="submit">

<Glyphiconglyph="off"/>Add

</Button>

</div>

}

</form>

</Modal.Body>

<Modal.Footer>

<ButtonbsStyle="danger"bsSize="default"

onClick={this.handleCloseModal}>

<Glyphiconglyph="remove"/>Close

</Button>

</Modal.Footer>

</Modal>

}

</div>

);

}

Thereisacapabilitytocommentoneachstory.Thatisdonethroughamodal

dialog similar to the one used for user registration. Here is the image of the

viewingofcomments:

Figure101-NewsWatcheraddcommentUI

Componentsupportingcode

ThiscodeisverysimilartotheNewsViewcomponent.Thedifferenceisin

thecodeforaddinganewcomment.Hereisthecode:

importReact,{Component}from'react';

importPropTypesfrom'prop-types';

import{FormGroup,ControlLabel,Button,Modal,Glyphicon,Media}from'react-bootstrap';

import{connect}from'react-redux'

importsuperagentfrom'superagent';

importnoCachefrom'superagent-no-cache';

import{FieldGroup,toHours}from'../utils/utils';

import'../App.css';

classSharedNewsViewextendsComponent{

constructor(props){

super(props);

this.state={

comment:"",

selectedStoryIdx:0

};

}

componentDidMount(){

if(!this.props.session){

returnwindow.location.hash="";

}

const{dispatch}=this.props

dispatch({type:'REQUEST_SHAREDNEWS'});

fetch('/api/sharednews',{

method:'GET',

headers:newHeaders({

'x-auth':this.props.session.token

}),

cache:'default'//no-storeorno-cache?

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

for(vari=0;i<response.json.length;i++){

response.json[i].story.hours=

toHours(response.json[i].story.date);

}

dispatch({type:'RECEIVE_SHAREDNEWS_SUCCESS',news:response.json});

dispatch({type:'MSG_DISPLAY',msg:"SharedNewsfetched"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`SharedNewsfetchfailed:${error.message}`});

});

}

handleOpenModal=(index,event)=>{

this.setState({selectedStoryIdx:index,showModal:true});

}

handleCloseModal=(event)=>{

this.setState({showModal:false});

}

handleAddComment=(event)=>{

const{dispatch}=this.props

event.preventDefault();

fetch(`/api/sharednews/${this.props.news[this.state.selectedStoryIdx].story.storyID}/Comments`,{

method:'POST',

headers:newHeaders({

'x-auth':this.props.session.token,

'Content-Type':'application/json'

}),

cache:'default',//no-storeorno-cacherodefault?

body:JSON.stringify({comment:this.state.comment})

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==201){

thrownewError(response.json.message);

}

varstoryIdx=this.state.selectedStoryIdx;

dispatch({type:'ADD_COMMENT_SUCCESS',comment:this.state.comment,

displayName:this.props.session.displayName,

storyIdx:storyIdx});

this.setState({showModal:false,comment:""});

dispatch({type:'MSG_DISPLAY',msg:"Commentadded"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`Commentaddfailed:${error.message}`});

});

}

handleCommentChange=(event)=>{

this.setState({comment:event.target.value});

}

render(){

...TAKENOUT...ALREADYSHOWN...

}

SharedNewsView.propTypes={

dispatch:PropTypes.func.isRequired

};

constmapStateToProps=state=>{

return{

session:state.app.session,

news:state.sharednews.news,

isLoading:state.sharednews.isLoading

}

exportdefaultconnect(mapStateToProps)(SharedNewsView)

17.7ProfilePage(src/views/profileview.js)

The profile page allows the user to create one or more news filters. Each

filterhasatitleandalistofkeywords.Youhavethesametypeofbuttondrop-

down you have seen before. Then you have the form and the three buttons to

save,delete,andcreateanewfilter.TheimageandHTMLareasfollows:

Figure102-NewsWatcherNewsFilterdialog

Thereisalinktoallowtheusertodeletetheiraccount.TheHTMLprovides

amodaldialoglikeyouhaveusedbefore.Hereistheimage:

Figure103-NewsWatcherunregisterdialog

render(){

if(this.props.isLoading){

return(

<h1>Loadingprofile...</h1>

);

}

return(

<div>

<h1>Profile:NewsFilters</h1>

<FormGroupcontrolId="formControlsSelect">

<FormControlbsSize="lg"componentClass="select"

placeholder="select"

onChange={this.handleChangeFilter}

value={this.state.selectedIdx}>

{this.props.user.newsFilters.map((filter,idx)=>

<optionkey={idx}

value={idx}><strong>{filter.name}</strong>

</option>

)}

</FormControl>

</FormGroup>

<hr/>

<form>

<FieldGroup

id="formControlsName"

type="text"

label="Name"

placeholder="NewFilter"

onChange={this.handleNameChange}

value={this.props.user.

newsFilters[this.state.selectedIdx].name}

<FieldGroup

id="formControlsKeywords"

type="text"

label="Keywords"

placeholder="Keywords"

onChange={this.handleKeywordsChange}

value={this.props.user.

newsFilters[this.state.selectedIdx].keywordsStr}

<divclass="btn-groupbtn-group-justified"role="group"

aria-label="...">

<ButtonbsStyle="primary"bsSize="default"

onClick={this.handleAdd}><Glyphiconglyph="plus"/>Add

</Button>

<ButtonbsStyle="primary"bsSize="default"

onClick={this.handleDelete}><Glyphiconglyph="trash"/>Delete

</Button>

<ButtonbsStyle="primary"bsSize="default"

onClick={this.handleSave}><Glyphiconglyph="save"/>Save

</Button>

</ButtonToolbar>

</div>

</form>

<hr/>

<p>NolongerhaveaneedforNewsWatcher?<aid="deleteLink"

style={{cursor:'pointer'}}

onClick={this.handleOpenModal}>DeleteyourNewsWatcherAccount</a>

</p>

<Modalshow={this.state.showModal}onHide={this.handleCloseModal}>

<Modal.HeadercloseButton>

<Modal.Title>Un-Register</Modal.Title>

</Modal.Header>

<Modal.Body>

<formonSubmit={this.handleUnRegister}>

<Checkboxchecked={this.state.deleteOK}

onChange={this.handleCheckboxChange}>

CheckifyouaresureyouwanttodeleteyourNewsWatcheraccount

</Checkbox>

<Buttondisabled={!this.state.deleteOK}bsStyle="success"

bsSize="lg"blocktype="submit">

<Glyphiconglyph="off"/>DeleteNewsWatcherAccount

</Button>

</form>

</Modal.Body>

<Modal.Footer>

<ButtonbsStyle="danger"bsSize="default"

onClick={this.handleCloseModal}><Glyphiconglyph="remove"/>

Cancel

</Button>

</Modal.Footer>

</Modal>

</div>

);

}

Componentsupportingcode

All the functions necessary to provide the functionality behind the button

clicks are made available and should look very similar to code you have seen

before.

importReact,{Component}from'react';

importPropTypesfrom'prop-types';

import { FormGroup, FormControl, Checkbox, Button, Modal, Glyphicon, ButtonToolbar } from

'react-bootstrap';

import{connect}from'react-redux'

importsuperagentfrom'superagent';

importnoCachefrom'superagent-no-cache';

import{FieldGroup}from'../utils/utils';

import'../App.css';

classProfileViewextendsComponent{

constructor(props){

super(props);

this.state={

deleteOK:false,

selectedIdx:0,

};

}

componentDidMount(){

if(!this.props.session){

returnwindow.location.hash="";

}

const{dispatch}=this.props

dispatch({type:'REQUEST_PROFILE'});

fetch(`/api/users/${this.props.session.userId}`,{

method:'GET',

headers:newHeaders({

'x-auth':this.props.session.token

}),

cache:'default'//no-storeorno-cache?

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

for(vari=0;i<response.json.newsFilters.length;i++){

response.json.newsFilters[i].keywordsStr=

response.json.newsFilters[i].keyWords.join(',');

}

dispatch({type:'RECEIVE_PROFILE_SUCCESS',user:response.json});

dispatch({type:'MSG_DISPLAY',msg:"Profilefetched"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`Profilefetchfailed:${error.message}`});

});

}

handleUnRegister=(event)=>{

const{dispatch}=this.props

event.preventDefault();

fetch(`/api/users/${this.props.session.userId}`,{

method:'DELETE',

headers:newHeaders({

'x-auth':this.props.session.token

}),

cache:'default'//no-storeorno-cache?

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

this.props.appLogoutCB();

dispatch({type:'MSG_DISPLAY',msg:"Accountdeleted"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`Accountdeletefailed:${error.message}`});

});

}

handleNameChange=(event)=>{

this.props.dispatch({type:'ALTER_FILTER_NAME',

filterIdx:this.state.selectedIdx,value:event.target.value});

}

handleKeywordsChange=(event)=>{

this.props.dispatch({type:'ALTER_FILTER_KEYWORDS',

filterIdx:this.state.selectedIdx,value:event.target.value});

}

handleOpenModal=(event)=>{

this.setState({showModal:true});

}

handleCloseModal=(event)=>{

this.setState({showModal:false});

}

handleChangeFilter=(event)=>{

this.setState({selectedIdx:parseInt(event.target.value,10)});

}

handleAdd=(event)=>{

const{dispatch}=this.props

event.preventDefault();

if(this.props.user.newsFilters.length===5){

dispatch({type:'MSG_DISPLAY',

msg:"NomorenewsFiltersallowed"});

}else{

varlen=this.props.user.newsFilters.length;

dispatch({type:'ADD_FILTER'});

this.setState({selectedIdx:len});

}

handleDelete=(event)=>{

event.preventDefault();

this.props.dispatch({type:'DELETE_FILTER',

selectedIdx:this.state.selectedIdx});

this.setState({selectedIdx:0});

}

handleSave=(event)=>{

const{dispatch}=this.props

event.preventDefault();

fetch(`/api/users/${this.props.session.userId}`,{

method:'PUT',

headers:newHeaders({

'x-auth':this.props.session.token,

'Content-Type':'application/json'

}),

cache:'default',//no-storeorno-cacherodefault?

body:JSON.stringify(this.props.user)

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

dispatch({type:'MSG_DISPLAY',msg:"Profilesaved"});

})

.catch(error=>{

dispatch({type:'MSG_DISPLAY',

msg:`Profilesavefailed:${error.message}`});

});

}

handleCheckboxChange=(event)=>{

this.setState({deleteOK:event.target.checked});

}

render(){

...TAKENOUT...ALREADYSHOWN...

}

ProfileView.propTypes={

appLogoutCB:PropTypes.func.isRequired,

dispatch:PropTypes.func.isRequired

};

constmapStateToProps=state=>{

return{

session:state.app.session,

user:state.profile.user,

isLoading:state.profile.isLoading

}

exportdefaultconnect(mapStateToProps)(ProfileView)

You can see that there is code there to limit the number of filters to five.

Whatdoyousupposewouldhappeniftheuserusedthebrowserdevelopertools

tomesswiththatcodeandthenaddedthousandsoffilters?Itcouldcreateavery

largedocumentsinMongoDB.BeawarethatyoucannoteverrelyonyourUI

side bounds-checking code to do the right thing, as it is subject to tampering.

Thus,youhavetoplacecodein yourservicelayerthatwillonlytakethe first

fivefilters.

Itisalsonoteworthy,howthefunctiontohandletheloggingoutisnotinthis

code.Instead,theAppComponentthatusestheProfileViewcomponentpasses

inafunctionasapropertyonthepropsandthenthecallbackisusedtogetat

codeinApp.jsforthat.Thisseemedtobemoreappropriateforthehighercalling

codetocontrolthatandnotneedtoduplicatecodeinthiscase.

17.8NotFoundPage(src/views/notfound.js)

TheNotFoundpageissomethingthatwouldbedisplayediftheusertriedto

navigateby changingthe URLin the browserto somepage thatdid not exist.

You can see this in action if you try and go to

https://www.newswatcher2rweb.com/#/blah.HereisthatComponentcode.

importReact,{Component}from'react';

import'../App.css';

classNotFoundextendsComponent{

render(){

return(

<div>

<h3>404pagenotfound</h3>

<p>Thepageyouarelookingfordoesnotexist.</p>

</div>

);

}

exportdefaultNotFound;

This completes the discussion of each of the files and their corresponding

controllercode.EverythingcanbezippedupanddeployedtotheAWSElastic

Beanstalkenvironmentandused.OnWindows,youwouldselectthefoldersand

files as shown before and then right click and select Send to->Compressed

(zipped)folder.

Chapter18:UITestingofNewsWatcher

With the NewsWatcher application code completed and running, you will

wanttothoroughlytestitwitheveryconceivablescenario.Manuallydoingthis

isgreatforyourentertainment,butitwillsoongrowtediousifyouhavethatas

your only way of finding bugs. For example, what happens if you make any

changestoyourcodebase?YouhavethefunctionandloadtestsoftheAPI,but

you still need to run the UI through its paces to exercise the React JavaScript

code. Not to fear, there are many great solutions to this problem and I will

presentthisaswellasgivegeneraltipsfordebuggingthecodeinChrome.

Therearemanydifferenttechnologiesbeingusedinthecodeandeachhas

techniques for testing it in detail. For example, the React Router usage and

Redux code can be independently tested. Each individual component can be

testedinwaysthatinstantiatethemandverifycorrectDOMelementsarepresent

with the correct attributes and properties. Then you can also try testing

frameworksthatdoanautomatedrun throughofthecompleteapplicationina

browser.YouuseIDsofeachUIelementtoidentifywhattoclickorinspect.

18.1UITestingwithSelenium

There are tools you can use to record your interactions within a browser

session and then play those back. Some tools record screen locations of your

clicksandthenrelyontheUIelementstobeinthatexactsamelocationlaterfor

theinteractiontowork.OthertoolsunderstandtheDOMandcanfindelements

youspecifytoclickon.

One tool you can use to do automation testing of a UI, is a tool created a

while a while ago named Selenium. It was built so long ago that it did not

support JavaScript directly. Luckily, someone did the work to write a node.js

modulethatexposesitscapabilities.

Thefirststeptowardsaccomplishingyour UIautomationtestingwillbeto

installthe“selenium-webdriver”fromNPMtohaveitlocalinyourproject.Be

awarethatthismightnotbeassimpleasitsounds.Payattentiontotheoutput

windowasyouinstallit.Itmightfailbecauseofdependencies.Ihadtoinstall

the JDK, Python tools, and even a more complete Visual Studio install with

someoftheC++tools.Iwouldnothavedreamedthosewouldberequired.Your

experiencemightbedifferentonadifferentOS.

You might also check out an NPM module named Protractor. This is

somethingwrittenontopofselenium-webdriver.Idecidedtogodirectlythrough

theselenium-webdrivermodule,sodidnotuseProtractor.

Theselenium-webdriverisjustanSDKtoaccessthebrowserDOMandthus

you still need some type of testing framework to organize and run your tests.

Mochaisperfectfordoingthis.Thisiswhatyouwouldrunfromthecommand

prompttogetyourUIautomationtestsrunning.

.\node_modules\.bin\mocha--timeout30000ui_automation_UAT.js

You will populate a Mocha test file with code that uses the selenium-

webdrivercapabilities.Youalsoneedtosetuptherequirestatementsforasserts

andformakingHTTPrequesttodosomeneededcleanupafteratestrun.Here

aretherequirestatements:

varassert=require('assert');

varwebdriver=require('selenium-webdriver');

varrequest=require('supertest')('http://localhost:3000);

YouwillusetheusualMochadescribeanditcodeblocks.Youneedtofirst

set up some code that runs before any tests in a before block. This code

initializes Selenium and sets up which browser you want it to use. The after

blockdoesthecleanup.

Usingthedriverobject,youcangetaccesstoanyelementintheDOM,to

inspectitoraffectitinsomeway.Forexample,withanHTMLtextcontrol,you

cansenditkeystrokestoentervaluesasauserwould.Buttonscanbeclicked,

etc.

Manytimes,youdosomethinglikeabuttonclickandthenneedtowaitfor

the UI to respond. The driver has a wait() function you use to wait for UI

elementstobeavailable.Forexample,ifyouwerenavigatingtoanewpage,you

woulduseawaittoblockuntilthepagewasready.Thewait()functiontakesasa

firstparameter,whatyouarewaitingfor.Youcanthuswaitforanelementtobe

visible. Inside that function, you provide the id that will be used by the

webdriverobjecttolocatetheHTMLelement.Thereisatimelimitsetforhow

longitshouldwait.Itwillwaituptothattime,butiftheelementappearsbefore

that, it will immediately move on. Sometimes you will need to put in a

setTimeout()calltodothedelay.Thisshouldbeavoidedastheseaddupandadd

tothetotaltimeittakesforatestruntocomplete.

Here is the specifying of an id in your HTML that you then can use to

identifyabuttonbylaterinyourtestcode.

<buttonid="btnRegister"ng-click="register()">Register</button>

Aswith all Mocha tests, you need to call done()when that test is finished

andmoveontothenexttest.

ThewholepointofputtingtogetheraUIautomationtestwithSeleniumisto

mimic what you would normally do manually and thus have a repeatable test

suitethatyoucanrunaspartofaCI/CDscriptandsaveyoualotoftime.

ThereistheconceptofaUserAcceptanceTestorUATthatbasicallyisthe

userscriptthatyouwanttofollowtoprovethattheUIcandoeverythingitis

supposed to do. Development teams can create a UAT for each code iteration

theygothroughbeforeadeploymentcanbeapproved.

Youcansetupthetesttogoagainstyourstaging,productionorlocalhosted

site.HereisasmallsampleofthecodefortheUIautomationtestingthatuses

Selenium.IhavesetituptouseInternetExplorer,butyoucanuseanybrowser

youlike,evenahiddenone.Youwillgetamessageifyoudonothavethedriver

installed for the browser you want to use. For example, with IE, the file is

IEDriverServer.exe. You will see a message on where to get it from. You just

unzipitandcopyittoalocationthatisinyourpath.

NewsWatcherSeleniumtests

Thiscodesimplylaunchesthesiteandthenclicksonthelogintabandthen

attemptstologinwithauseremailthatdoesnotexist.Thecodeteststhatthe

correcterror message appears. Note how there needs to be delays put into the

testcodetowaitforUIchangestohappen.

describe('NewsWatcherUIexercising',function(){

vardriver;

varstoryID;

vartoken;

//Runsbeforealltestsinthisblock

before(function(done){

driver=newwebdriver.Builder().withCapabilities(

webdriver.Capabilities.ie()).build();

driver.get('http://localhost:3000');

driver.wait(webdriver.until.elementLocated(

webdriver.By.id('loginLink')),10000).then(function(item){

done();

});

//Runsafteralltestsinthisblock

after(function(done){

driver.quit().then(done);

});

it('shoulddenyaloginwithanon-registeredemail',function(done){

driver.findElement(webdriver.By.id('loginLink')).click();

driver.findElement(webdriver.By.id('formControlsEmail2')).

sendKeys('T@b.com');

driver.findElement(webdriver.By.id('formControlsPassword2')).

sendKeys('abc123*');

driver.findElement(webdriver.By.id('btnLogin')).click();

driver.wait(webdriver.until.elementLocated(

webdriver.By.id('currentMsgId')),5000);

//WaitafewsecondsfortheUItoupdate

setTimeout(function(){

driver.findElement(webdriver.By.id('currentMsgId')).getText().then(

function(value)

{

assert.equal(value,

'(Signinfailed:Error:Userwasnotfound.)');

done();

});

},5000);

});

Note:If you look at the selenium test code in my GitHub project, you will

find most of it commented out. This is because the code was taken from an

original Angular application and I decided not to continue with the Selenium

testingandinsteadgowithEnzyme.Seleniumstillhasitsplaceandisusefulfor

end-to-end user acceptance testing. You might want to investigate the use of

Nightwatch,asanicewrapperontopofSelenium.

18.2UITestingwithEnzyme

Enzyme is an API that you use to test your React components at the code

level. With it you will do component level testing. This is analogous to Unit

testingofservicelayercode.Youcanisolateagivencomponentandverifythat

itworks as an individual piece of code and then have a greater assurance that

whenconsumed,itwillallworktogether.

Youwillwanttostartwithtestsforcomponentsatthelowestlevelsbecause

itissometimesbesttousecodetotestcodeasitisthemostefficientmeansof

trying all the combinations of code paths (including error paths). As part of

doing this, you will mock data that might be coming from things like HTTP

calls.Thepointistoisolateacomponentasmuchaspossible.

Youcanalsotestcomponentsthatareatahigherlevel,suchastheonesthat

consume other components. This means you could even test the complete

applicationfromthehighestAppcomponentandexerciseallpartsofit.

ItisuptoyoutodecidehowrigorousyouwanttheEnzymeteststobe.For

example, you can test components and verify that when they are instantiated

theyhavetheproperstate,propsandstyles.Youcangofurtherandcodeuptests

foryour Redux reducers. Youcanalso do full testingas if you are interacting

withacomponentwithmouseclicksetc.

Node: Mocha is often used as the test runner for the Enzyme tests. I have

chosenJestinstead(builtontopofJasmine),becausethatiswhatisinstalledby

default with the React application that was created. It also has some great

featuresthatMochadoesnothave,suchasbuilt-incodecoverageandparallel

runningoftests.Youcan’treallytellthedifferenceanywaybetweenamochatest

fileandaJestone.Theybothhave‘describe’and‘it’blocksofcode.Bothalso

havetheabilitytousethedone()functiontodoasynchronoustesting.

EnzymeistheAPIyoumakeuseofinsideofeachtestcaseincodethatJest

willrun.Enzymeiseasytouse.Withit,youinstantiateacomponentwitheither

theshallow()ormount()functions.ShallowdoesnotcreateavirtualDOM,but

is lightweight and just creates the component at that level with no hierarchy

underneathit.Thisworksinmostcasesforyourtestingpurposes.

IfyouaretestingsomethinglikeacomponentthatiswrappedbyRedux,you

needtousethemount()capabilityasitwillgetlifecycleeventsrunning.Itwill

have a full virtual DOM, and a redux state store will be used. I will give

examplesofboththeshallowandthemountusage.

NewsWatcherEnzymetestswithshallow

Think back to the home page UI that had news stories served up. In that

React code, you can find a componentDidMount() function that goes to the

backendtofetchthelistoftopnewsstories.Thedatareturnedfromthefetchcall

getsplacedintothelocalcomponentstateandthentherender()functionupdates

theUIwiththelatestdata.Hereissometestcodeforverifyingallofthat,using

somemockeddata.Iplacedalotofcommentsinthecodetohelpexplainit.

importReactfrom'react';

import{shallow}from'enzyme';

import{createStore}from'redux';

importreducerfrom'../reducers';

importHomeNewsViewfrom'./homenewsview';

//AhelperfunctiontoputtogetheranHTTPresponseforourmocking

constmockResponse=(status,statusText,response)=>{

returnnewwindow.Response(response,{

status:status,

statusText:statusText,

headers:{

'Content-type':'application/json'

}

});

};

describe('<HomeNewsView/>(mockeddata)',()=>{

it('newsstoriesaredisplayed',(done)=>{

//Thisisthepayloadthatourmockingwillreturn

constmockData=[{

contentSnippet:"ThelaunchofanewrocketbyElonMusk’sSpaceX.",

date:1514911829000,

hours:"33hoursago",

imageUrl:"https://static01.nyt.com/images/blah.jpg",

link:"https://www.nytimes.com/2018/01/01/science/blah.html",

source:"Science",

storyID:"5777",

title:"RocketLaunchesandTripstotheMoon",

}];

//Weusetheactualreduxstorewithourofficialreducers

//WereplacetheJavaScriptfetch()functionwithourown

//andalwaysreturnourmockdata

conststore=createStore(reducer)

global.fetch=()=>Promise.resolve(mockResponse(200,null,JSON.stringify(mockData)));

//HereistheusageofEnzymetoinstantiateourcomponent

constrc=shallow(<HomeNewsViewdispatch={store.dispatch}/>,{disableLifecycleMethods:true})

expect(rc.state().isLoading).toEqual(true);

//Wearedoingshallowinstantiation,soweneedtocallbyhand

//thecomponentDidMount()functionandwaitonthepromiseresolve

//tothendotestingagainstwhatisexpectedbecauseofthe

//newsstoryfetchandrenderwiththatdata

rc.instance().componentDidMount().then((value)=>{

//Allstatepropertieswillbeupdatedbynow,

//however,therendermaynothavehappenedyet.

//Theupdate()callismade,sothetestofthe<h1>element

//willnowhavetherefreshedvalue

rc.update();

//Verifysomeofthestatethatwasset

expect(rc.state().isLoading).toEqual(false);

//verifysomeactualelementsthatwererendered

expect(rc.find('h1').text()).toEqual('HomePageNews');

constlistNews=rc.state().news;

expect(listNews.length).toEqual(1);

expect(listNews[0].title).toEqual('RocketLaunchesandTripstotheMoon');

expect(rc.find('b').first().text()).toEqual('RocketLaunchesandTripstotheMoon');

//VerifytheReduxstatewassetandthereis

//asuccessfulmessageonfetchcompletion.

expect(store.getState().app.currentMsg).toEqual('HomePagenewsfetched');

done();

})

});

You can see the use of shallow() and then the further usage of the object

returnedwithfunctionslikefind()andstate().Theseallowyoutointerrogatethe

instanceforwhatisexpected.

The expect() function of Jest is used for the validations. The component

returnedfromtheshallowusagecanbeusedtodothingslikefindsub-elements

andinspectthem.Youcanseewherewefindanh1elementandverifythetext

ofit.Youcanalsoinspectthestatepropertiesofthecomponent.Thisisdoneto

verifywhatnewsstorieswerefetched.

AnothertestisdonethatinspectstheReduxstorewithagetState()call.You

canlookatanythinginthestoreasyouseeishappeningintheverificationof

whattheapp.currentMsgissetto.

Since we are doing shallow rendering, the componentDidMount() function

willnotbecalledaspartofthelifecycleoperations.Thatiswhyyouseeitbeing

called in test code. The call to componentDidMount() is asynchronous, so we

neededtomakeuseofthedone()functiontotellJestthatthistestiscomplete.

Note:YoumightbetemptedtouseasetTimeout()intestcodeanddelayfora

fewsecondsuntiltheUIhasrendered.ThisactuallywillnotworkinJest,andis

abadideaanyway.Ifyouhadhundredsofteststhateachhaddelays,youwould

increasethetime ittakesto runyour testsandcould notreallybeguaranteed

anyraceconditionswouldworkout.

TheactualcomponentDidMount()componentcodegetscalled,soitwilldo

thingslikemakethefetchcallandthentryandalsosetthestateinReduxwitha

dispatchcall.Sincewewanttocontroltheactualreturnedresponse,weneedto

mockallofthisupandoverridethefetchfunction.WealsosetupReduxhere,

becausethatisnormallydoneintheAppcomponent,andweareworkingata

levelbelowthat.

Note:AnalternativetocallingthecomponentDidMount()wouldbeforyou

tomimicthecallsthatarehappeningtotheReduxactions.Youcouldplacecalls

tostore.dispatch()andthendosomeexpect()calls.Youcouldalsosetthestate

orpropsdirectlyoncomponentsandthenforceanupdateandthendotheexpect

tests.

NewsWatcherEnzymetestswithmount

The code to test the login must be a bit different. This is because the

componentisneedingRedux.Inthiscase,weneedtousethemount()capability

of Enzyme. We have to use mount() because we want to use the connected

component.Ithasthefunctionalitytousethereduxdispatchtoupdatepropsand

wecanverifythatthestorestateiscorrectlyset.

Hereissomecodethatmountsthecomponentandthenteststhataloginwas

successful.

it('Usercanlogin',(done)=>{

constmockData={

displayName:"Buzz",

userId:"1234",

token:"zzz",

msg:"Authorized"

};

conststore=createStore(reducer)

global.fetch=()=>Promise.resolve(mockResponse(201,null,

JSON.stringify(mockData)));

//Needtomockthelocalstoragecallaswell

constlocalStorageMock={

setItem:()=>{},

removeItem:()=>{}

};

global.localStorage=localStorageMock

constwrapper=mount(<ConnectedLoginViewstore={store}/>)

letrc=wrapper.find(LoginView);

expect(rc.props().session).toEqual(null);

rc.instance().handleLogin({preventDefault(){}}).then((value)=>{

expect(store.getState().app.session.displayName).toEqual('Buzz');

expect(store.getState().app.currentMsg).toEqual('SignedinasBuzz');

done();

})

});

Weoverridefetchagainasbefore.Noticehowwealsoneededtoprovidea

stubbed out preventDefault() call to not do anything, as handleLogin() is

expectingthatfunctiontoexist.

Note: We could have caused a click to have happened on the login button

and the code would have proceeded to execute that. The problem is that we

wouldnotknowwhenthatfinished,sowecansimplycallthefunctionourselves

andwaitfortheresolveanddowhattestingwewantafterthat.

Youwill notice we don’t even fill in the email and password, as our fetch

mockingdoesnotpayattentiontoanythingpassedinandreturnswhatwewant,

underourcontrol.

UIelementinteraction

Hereissomecodethatdoessomemanipulationoftheactualelementslike

thetextcontroltoholdtheemailandthecheckboxcontrol.Youcanuseshallow

ormount andaccomplish thesame thing. Whatwe are reallychecking hereis

thebindingofeachcontrolbylookingatthestatepropertiesthatshouldbesetas

aresultoftheinteractions.

it('Usercanchangeremembermecheckboxandenteranemail',()=>{

constrc=shallow(<LoginViewdispatch={()=>{}}session={null}/>,{disableLifecycleMethods:true

})

expect(rc.state().remeberMe).toEqual(false);

rc.instance().handleCheckboxChange({target:{checked:true}});

expect(rc.state().remeberMe).toEqual(true);

rc.find('#formControlsEmail2').last().simulate('change',{target:{value:"abc@def.com"}})

expect(rc.state().email).toEqual("abc@def.com");

});

Inthecaseofacheckbox,thereisnotawaytoclickitorcauseachangetoit

to have it run the onChange() handler so we just call the

handleCheckboxChange()methodourselves and fakewhat wouldbe passed in

fortheeventparameter.

Fortheemailtextinputelement,wecansimulatethetextentry.Inthiscase,

wecangetanonChange()tohappenautomaticallyforus.

GettingsetuptouseEnzymeandrunningtests

BeforeusingEnzyme,youneedtoinstallthenpmpackagesasfollows:

npminstall--saveenzymeenzyme-adapter-react-16react-test-renderer

ThenyoualsoneedtocreateafileinthesrcdirectorynamedsetupTests.js

withthefollowingcontents:

//src/setupTests.js

import{configure}from'enzyme';

importAdapterfrom'enzyme-adapter-react-16';

configure({adapter:newAdapter()});

Torunthetestsyoujustrunthefollowing:

npmruntest-react

Youcanalterthetestscriptinthepackage.jsonfiletoaddtherunningofthe

Enzymetestsandalsoaddaflagtotellittoincludecodecoveragenumbers.

"test": "mocha --timeout 30000 test/functional_api_crud.js && react-scripts test --coverage --

env=jsdom",

Ifyoudosomesearchingonlineandyouwillfindallkindsofcreativeways

to test with Jest and Enzyme. For example, I did not mention the ability of

EnzymetocompareDOMsnapshotsfromonetestruntoanother.

Goodtestscodecansometimesbeasdifficultorevenmoredifficultthanthe

actualproductcode.Ihaveworkedonseveralprojectswheretherewastwiceas

muchtestcodewrittenasproductioncode.

Codecoveragenumberslookasfollows:

Figure104-F12debuggerconsole

18.3DebuggingUICodeIssues

Tolocallydebugyouserversidenode.jscodeisverysimple.Byusingthe

VS Code editor, you have the capability to run it locally and set breakpoints.

Reactcode,however,runsinthebrowser.Browsershavetheirowndebugging

capabilities that you must learn and make use of. For example, with Chrome,

thereisaselectioninthemenuunderMoretoolstolaunchDevelopertools.

Onceopen,youcanclickontheConsoletabandseeerrorsthathappenin

yourcode.Thisimageshowsacodeerrorforanundefinedproperty:

Figure105-debuggerconsoleinChrome

This window is very handy to have open, to see problems you might not

otherwisenotice.Manytimes, itisquite easytogo andfixthe bugsyou find.

Other times, you will need to step through the lines of code to tell what is

happening.SimplyclicktheSourcestabandyoucanbrowseyourcodeandset

breakpoints in the browser. The experience is typical of any debugger, in that

youcaninspectvariablesandsetwatchesonthem.

ThisnextimageshowshowIusedthedebuggertosetabreakpointandthen

inspectedtheresponsedatabeingreturnedfromanHTTPrequest.Theleftpane

hasalistoffilesthatyoucanlookthrough.IopeneduptheJavaScriptcodefor

thehomenewsviewing.Inthepanewherethecodeis,youcanclickonaline

numberandsetabreakpoint.Youmayneedtohitthebrowserrefreshbutton,or

clickaroundinyourapplicationtomakethecoderun,tohityourbreakpoint.

Figure106-NewsWatcherHTTPrequestdebugging

YoucanhaveVSCoderunningthenodeserverandplacebreakpointsinthe

serversidecodeanddebugbackandforthinbothenvironments.

If you have written HTML, you know the capabilities of the browser

developertoolsandwillcertainlyknowhowtoinspecttheDOMelementsand

theirassociatedCSS.ClicktheDOMExplorertabtotrythisout.Forexample,

youmightfindthatanelementisnotstyledorfunctioningasexpected.Aquick

inspectioncanusuallyturnuptheissue.

TheNetworktabshowsyoutheHTTPtrafficgoingbackandforthsimilarto

what you see using Fiddler or Postman. The Performance tab can let you

capturethecallsgoingonandthenviewthatinacallgraphthatshowsthetime

ofeverythingindetails.Thisallowsyoutocheckforandfixperformanceissues.

TheMemorytabletsyousetuparunthatcapturesmemorysnapshotssoyou

caninvestigatememoryleaksinyourcode.

Figure107-ChromeNetworkcapturing

Chapter19:Server-SideRendering

The NewsWatcher application has been developed as a purely client-side

SPAapplicationthatloadsandrendersonaclientdevice.Itisruninthecontext

ofabrowser,whetherthatisonamobilesmartphone,tablet,laptoporpersonal

computer.Thisworksquitewell.

A SPA like NewsWatcher has great performance and mostly minimizes

traffic back and forth to just the data fetching. All the UI is always rendered

client-sideandisneverloadedfromtheserver,onceitisallbroughtover.There

isabitofresistanceinthedevelopmentcommunitytofullysanctionSPAsasthe

waytogo.Theirobjectionsarebasedontwopointsasfollows:

1. Performance is better servicing up pages from a server. A server

renderedappcancollectallthedataontheserversideandserveitup

fasterthanaSPA.Especiallywhenitcomestotheinitialpageloading,

ifthereisalotthatgoeswithit.

2. SEO (Search Engine Optimization) is more efficient for server

renderedHTMLpagesbecauseofhowGoogleindexesthem.

What some people are advocating, is to move to an SSR (Server Side

Rendering)designtoaddressthesetwoissues.Therecan,ofcourse,beahybrid

solutiontogiveyouthebestofaSPAandanSSRsolution.Let’sfirsttakealook

attheobjectionsonebyone.

Asfarasthefirstpoint,thiscanbemitigatedbyimplementingcodesplitting

so that not everything is sent to the client at the start. This is where you only

sendupUIasneeded.Thecodeiseffectivelysplitupintoparts.TheinitialUI

codeissentandrendersveryfastandthenotherpartsarebroughtinasneeded.

ThiswouldthenallowjustasfastofapageloadforaSPAasforanSSRsite.As

younavigatethroughtheUI,pagesarebroughttotheclientjustintime.

ThesecondpointisalsodebatableinthatGooglewillcrawlSPAapplications

andcanevenlookthroughJavaScriptcodewhiledoingwebindexing.

Note:ToreallydojusticetoanSSR,youalsoneedtoputinmetatagsinthe

HTMLashelpersforthewebindexerstohelpwithsearchengineoptimization

(SEO).

Ifyoudecideto,youcanwritethesitetobecompletelyrenderedasanSSR

site. You can even have Redux and React Router used on the server. As a

compromise,youcanmakeakindofhybridimplementation,wheresomeofthe

UIisrenderedontheserversideandreturnedasHTML/JavaScripttotheclient

browser and once there, that code can also contain React code and have that

code run on both the Browser side as well as on the SPA. This is what a

Universal application design pattern is all about – being able to run the same

typeofcode(ReactRouter,Redux,etc.)ontheserverandontheclient.

ThehybridUniversalJavaScriptsolutionwouldcertainlyaddressbothissues

andmightbewhatyouwanttogotowards.Thiswouldmeanthatyouhavethe

initial page rendered with SSR, including its Redux store data along with the

initialpage so that it renders instantly. This gives you the speed for the initial

pagerendering.TheSEOissuewouldbe takencareofforthisinitialpage for

Googletoindex.Thepageloadsimmediatelyanddoesnothavetogobackto

fetchanyadditionaldatauntilfurtheruserinteractionsarehappening.

AUniversalhybridapproachisgreatifyouhaveaninitialroutepagewith

criticalinformationtobeindexed.Fromthere,thatpagecanevenfunctionasa

SPA,withsub-navigationthatgoesalongwithit.Youcandecidewhatpagesyou

want served up as SSR routes and then have each of those be SPA pages

functioningontheirown.

Note:ifyouarethinkingthatalotofyouruserswillbeonmobiledevices

that have cellphone or Wi-Fi connection anyway, you might as well develop

native applications. This means you have a purely native application they

download. This is in essence functioning like the SPA does, except that the

application is always instantly there on the device. You also get the benefit of

writingthenativeapplicationtoworkinamodewhereitisoff-line.Youcanalso

investigatewhataProgressiveWebapplicationisfordoingthatinthebrowser.

Youwillalsowanttoimplementsomecachingontheserversideifyouare

doing SSR. This is a huge performance advantage. For example, if the home

newspageisgoingtostaythesameforeveryoneforafewhours,yourenderit

andservethatupfromacachedcopyandthenupdateitwhenanewsetofnews

isreadytogo.ThereareNPMmodulesyoucanusethatmakethisseamlessand

areeasytoputintoyourcode.

19.1NewsWatcherandSSR

Idecidedtoexperimentwithhavingtheinitialhomenewspageasaserver-

side rendered page and then the rest of the application being a client SPA. It

turnedouttoworkreallywellandisanoptionforthosethatdecidetogothis

route. It may even turn out that someday create-react-app generates code that

allowsyoutheoptionofdoingcodeasSSR.

Note:YoucanfindmySSRexperimentonGitHub.Thiswasabriefattempt

andshouldnotbetakenasafinishedprojecttopatternanythingafter.Iwasable

to preserve the original app without doing an eject

(https://github.com/eljamaki01/NewsWatcher2RWebSSR). Be aware that I took

thecodesnapshotforthisexperimentatanearlystageofNewsWatcher,soalot

changedbetweenthatandtheotherrepositorythatendedupbeingthemainone

forthisbook.

What I did, is set things up for the route to the home news page to be

rendered with SSR. The rest of the application still works the same as a SPA.

TheideaisthataninitialcallcomesinfortheNewsWatchersiteandanHTML

page is rendered on the server side for it. This initial page is the home news

page. That HTML can also contain a special section that contains all of the

actualfetched data for the newsstories. This also goes up. Youmightwonder

whyintheworldthatwouldbegoingon.TheanswerhastodowithhowReact

works.

ReactwillonlyrenderUIthathaschanged.Thus,aninitialpageofthehome

newsisrenderedandthenReactalsodoestheworktoworkasifitisaSPAand

takesthedatasentupandalsotriestocreateavirtualDOMforthehomenews

fromthat.Itturnsouttobeidenticalwithwhatwasalreadybeingshowninthe

actualDOMandsoitthrowsitaway.

It turns out that the page instantly loads and there is no delay. It all just

instantlyappears.Otherviews,suchastheusernewspagewillshowtheloading

text while the componentDidMount() code chunks away, but the home news

pagenevershowstheloadingtext.

Onechangethatneedstohappenisthattheserver.jsfileneedstoserveup

theroute forthe main‘/’ routethat isthe UIapplication itself.That codewill

end up being in a new file that you create named ssrrender.js. Here is the

server.jsfilecodetoreturntheserver-renderedmainpage:

process.env.BABEL_ENV='production';

process.env.NODE_ENV='production';

require('babel-register')({

ignore:/\/(build|node_modules)\//,

presets:['env','react-app']

})

constSSRRender=require('./ssrrender');

constSSRRenderRouter=require('./ssrrenderRouter');

app.use('/',SSRRenderRouter)

//ServingupofstaticcontentsuchasHTML,images,CSSfiles,andJavaScriptfiles

app.use(express.static(path.join(__dirname,'build')));

app.use('/',SSRRender)

Thecodetorenderthestaticpageisgoingtolookfamiliar.Youwillevensee

the use of the Provider component and the Redux store and reducers. It is all

there as before. This means that the exact process of rendering a page on the

clientis now happening on theserver.Even the bootstrap style filesare there.

There are two interesting lines that do string replacements on the HTML

template. The first one does the actual replacement of the HTML that was

renderedbyReact.ThesecondoneistheonethataddstotheReduxstate.Itis

transferred with the news story list up to the client to be used there. This is

becausetheComponenbtDidMount()isnotusedtoretrieveitanymore.Hereis

allofthecode:

//ssrrender.js

constpath=require('path')

constfs=require('fs')

constconfig=require('./config');

constReact=require('react')

const{renderToString}=require('react-dom/server')

const{StaticRouter}=require('react-router-dom')

const{createStore}=require('redux')

const{Provider}=require('react-redux')

const{default:App}=require('./src/App')

const{default:reducer}=require('./src/reducers')

import{toHours}from'./src/utils/utils';

varexpress=require('express');

import'bootstrap/dist/css/bootstrap.css';

import'bootstrap/dist/css/bootstrap-theme.css';

varrouter=express.Router();

//ReturnalltheHomePagenewsstories.Verifywehavealoggedinuser.

module.exports=functionhandleSSR(req,res,next){

req.db.collection.findOne({_id:config.GLOBAL_STORIES_ID},

{homeNewsStories:1},

function(err,doc)

{

if(err)returnnext(err);

//Populateaninitialstate

for(vari=0;i<doc.homeNewsStories.length;i++){

doc.homeNewsStories[i].hours=toHours(doc.homeNewsStories[i].date);

}

letpreloadedState={homenews:{isLoading:false,

news:doc.homeNewsStories}}

//CreateanewReduxstoreinstance

conststore=createStore(reducer,preloadedState)

constcontext={}

consthtml=renderToString(

<Providerstore={store}>

<StaticRouter

location={req.url}

context={context}

<App/>

</StaticRouter>

</Provider>

)

//GrabtheinitialstatefromourReduxstore

constfinalState=store.getState()

//Sendtherenderedpagebacktotheclient

constfilePath=path.resolve(__dirname,'build','index.html')

fs.readFile(filePath,'utf8',(err,htmlData)=>{

if(err){

console.error('readerr',err)

returnres.status(404).end()

}

//We'regood,sosendtheresponse

letreplace1=htmlData.replace('{{SSR}}',html)

letreplace2=replace1.replace(`console.log("REPLACE")`,

`window.__PRELOADED_STATE__=${JSON.stringify(finalState).replace(/</g,'\\u003c')}`)

res.send(replace2)

})

});

};

Thereisoneminortweaktothehomenewsview.jsfile.Yousimplycomment

outthecomponentDidMount()function,becauseitalreadyhasthenewsstories

intheinitialloadandthatisplacedinthereduxstoretobeused.

The file index.js also has a minor tweak where it uses a special window

objectpropertywindow.__PRELOADED_STATE__propertythathasthenews

storiesandloadsthatintotheReduxstoreontheclientside.Youseethatonthe

server-sidecodeinssrrender.jsthatthestoreisalsopreloadedwiththatdata.You

canlookthereandyouwillfindtheactualcalltothedatabase.ThereisnoHTTP

call,asontheserversidewehavethedatabaseconnectiontousedirectly.

Note:Server-siderenderingisnotforeveryone.Whateveryoudo,youwould

havetoweighthecostsversusthebenefittohelpyoumaketherightdecision.It

canbetrickytogetrightandcancertainlymakeyourcodemorecomplicatedto

understandgoingforward.

Chapter20:NativeMobileApplication

developmentwithReactNative

The existing NewsWatcher React SPA client code can be taken and

transformedintocodethatrunsasanativeapplicationonamobiledevice.The

ReactWebappwasbuilttoberesponsive,meaningadapttothescreensizethat

it is running on. It was built using Bootstrap to give it such things as a

collapsiblemenuwhenrunningonamobilephoneinthebrowser.Thisstilldoes

notcomparetowhatperformanceandUIcapabilitiescanbeaccomplishedwith

atrulynativeapplication.

OnebenefitofNativeapplicationsisthattheycanbedesignedtoruninan

off-linemodewhennodataorWIFIconnectionisavailable.Sure,thereisthe

possibilityof doing a progressive web application, but why notjust stick with

thereal thing and go completely native? Native applications have much better

performancethanabrowser-basedapplicationrunningonaphone.Theotherbig

differentiatoristhatNativeapplicationscanaccessallthehardwarefeaturesof

the phone itself, such as the camera, contacts, gyroscope and many other

features.

Yousimplydoabitoftweakingtothecodeintherenderfunctionsandthen

useatooltocreateanativeapplicationfilethatisappropriateforeitherGoogle

PlayorfortheAppleAppstore.

AReactNativeapplicationrunsthelow-levelmobileplatformcodethrough

a JavaScript “bridge” layer to interact with the phone for accessing all its

functionality and presenting a UI. There is a JavaScript engine interface that

existsonbothiOSandAndroid.Thisisalow-levellayerthattheninteractswith

theOSforthephonetodothesamethingsthatotherplatformlanguagesonthe

phonesdo.YoumayhaveheardofObjective-C,Java,XcodeorSwift.

React web sites write to the HTML DOM, as that is the layer for all

browsers, but for phones, it is the JavaScript engine layer as the point of

interaction.Thismeansyougetthemostperformantapplicationspossible,and

accesstoallthephonesOSandhardwarefeatures.

ThegreatbenefitwithReactNativeasthecoretechnologyisthatthecode

can be very similar to that of your React web application. This is because the

React Native library is directly related to the web version. You have all the

familiar concepts, like the usage of JSX, lifecycle events, Component class,

rendermethod,usageofmoduleslikeReduxandmanyotherReactcapabilities.

TheonebigdifferenceyouwillrunintoisthatyouhavenoHTMLelements

availableinReactNative.No<div>,<span>,<ul>etc.Thereisalsoadifferent

approach to CSS for styling. All of this is kind of a pain, because you would

havegonethroughalotofworktogetallofthatwrittenupinyourReactWeb

application. It does not turn out to be too much of a hassle though. Partly

because you really should re-think your UI design when you are writing an

applicationforasmallmobiledevicescreen.Thewebsitepagesinmanycases

donotmakesensetoberenderedthesameonamobiledevice.Evenifyouare

doing a responsive design with libraries like Bootstrap, that is still not good

enough.

For now, the goal should be to take advantage of what you can from your

ReactWebapplication.Thenyoucanstrivetowriteyourmobilecodeonceand

beabletoruntheapplicationonbothiOSandAndroid.Itisnotalwayspossible

towritethecodeandrunitonbothiOSandAndroid,butyoucancomecloseif

youworkatit.

Note:Thisbookdoesnotcoverthetopicofhowtointegratelow-levelnative

codeintoyourapplication.Thisisatechniquewhereyouactuallywritecodein

Objective-CforiOSorJavaforAndroid.NewsWatcherdidnotrequirethistobe

done, and you would need to understand when and how you would need this

advancedtechnique.

20.1ReactNativestarterapplication

TheReactNativeapplicationrequiresacertainamountofscaffoldingtobe

inplaceinordertobeabletobebuiltandrun.Asoneoption,youcouldcopya

projectfromsomeoneelsefromGitHub.Therearealsotoolstogenerateastarter

application.ThesecondchoiceiswhatIwentfor.

WhatIdidwastorunacommandtogetmyReactNativeapplicationcreated

and then I brought over code from the web application version. I was able to

provehoweasyitistotaketheReactwebsitecodeandwriteanativeapplication

fromthat.Certainly,mostofthecodelogicisgoingtobethesame.Itismostly

theUImarkupthatwouldhavetochange.IwillnowgothroughallthestepsI

tooktogettheReactNativestarterapplicationupandrunning.

Step One: Install a command line utility tool and a Desktop

DevelopmentTool

With Node installed, you could use NPM to install the create-react-native-

app command line utility. This is used to create a starting point application.

Anotheroption would be to use the Expo npm utility that you can also install

fromnpm.IwilluseExpo.Expoisfullysupportedandevenmentionedonthe

officialReactNativesite.

HereisthecommandtogetExpoinstalled:

npminstallexp--global

The exp utility will be used to generate your starter application and

eventually create the finished application file that will be uploaded to the app

storesforotherstoinstall,suchasinGooglePlay.

Youwillalsoneedtoinstalladesktopapplicationthatactsasadevelopment

environment for Expo. It is called Expo XDE (Expo Development

Environment).ToinstallXDEyoucangototheexpowebsiteanddownloadthe

versionyouneed.TherearedownloadsdependingontheOSofyourmachine.

StepTwo:Createthestarterapplication

NowyoucancreatetheReactNativeapplication.Todothat,youwillusethe

Expo utility you just installed. Open a command prompt and create a folder

where you want the application to be created in. Then run the following

command,replacingthetextwithwhatyouwantyourapplicationcalled:

expinitmy-new-project

Alternatively,youcanlaunchtheXDEdesktopapplicationandfromamenu

selection, create an application. This is actually what I did. I selected the

templatethatcreatesanapplicationwithsomenavigationUIalreadyinplace.

Note: When you run XDE you will need to provide a username and

password. This is because XDE requires an account to be used for the

generationofthenecessaryfileforeventualuploadingtoanappstore.

StepThree:Runtheapplicationonyourmobilephone

WiththeXDEapplicationopen,youcanselecttohavetheapplicationyou

build to be run on your phone. Before doing that, you need to first configure

your phone for tethering and actually have it connected via USB to your

computer.

To set up tethering and debugging on an Android phone you can do the

following:

GotoSettingsandunderGeneralyoucanscrolluntilyoufindtheselection

forAboutphone.OpenthatandthentaponSoftwareinfo.TapBuildnumber

(youmayhavetotapitmultipletimes)andthenyouwillseeamessagethatyou

haveenableddevelopermode.

NowgototheGeneralsettingstotheDeveloperoptionsselection.Select

USBdebuggingtoturniton.

Youareallsettomakechangestosomecodeandexperimentandseewhat

you can do with React Native. I found that with my Windows machine, there

weretimesthattheconnectiontomyphonedidnotseemtoworktoowell.For

me,the“localhost”selection,fromthesettingsworkedthebestastheprotocol.

Figure108-Settingtheconnectionprotocol

Now,youcangototheXDEUIandopenyourprojectandthenyouwillbe

abletoclickontheDevicebuttonandselectOpenonAndroid.Theapplication

willloadandbestartedonyourphoneandyoucaninteractwithit.

Ifyourecall,therewerecommandlinescriptstoputtogethertheapplication

bundlefordeployment.WiththeusageofExpoXDE,allofthathappensasyou

openaprojectandclicktohaveitrunonyourphone.

Note: You can start up a phone emulator on your PC that can run the

applicationinit.Thisallowsyoutodothisinplaceofusingyourphysicalphone

that is connected via USB. You can create virtual devices for many different

typesofphones,includingiPhones.Todothatyouneedtoinstallsomethinglike

Xcode,AndroidStudio,ortheGenymotionemulator.

Thefollowingscreenshowsanentryforthetetheredphonethatyouwould

select to have your application run on. If you had an emulator running, the

emulator would also show up. You can also set up configuration to have the

applicationauto-loadanycodechangesthatarebeingmadeintheeditor.Thisis

calledhotreloading.

Figure109-ExpoApplication

20.2Components

React Native has a set of components you can use for doing things like

displaying text, getting input and displaying images. Besides this, there are

plenty of UI component libraries that you can install from NPM that can give

youwhatyouarelookingfor.Forexample,ifyoulikeMaterial-UIfromGoogle,

foryourlookandfeel,therearelibrariesthatimplementthatforReactNative.

Youcannowlearnaboutafewofthesimplecomponentsavailableandsee

thattheyareusedlikeanyothercomponentthatyouhaveseen.Youusethemin

arenderfunctionandpassneededpropstothemintypicalReactcodeyouhave

seenbefore.

TheViewandtheTextcomponentsaresomeeasycomponentstostartyour

learningwith.ViewisbasicallywhatadivwasinHTML.Itisacontainerfor

othercomponentstohelpwiththeUIlayout.

TheText component is for displaying text. The following code is close to

whatyougetwhenyoucreateaReactNativeApplicationfromExpo.Youcan

findafilenamesApp.jsthatisthestartingpointforrenderingyourUI.

importReactfrom'react';

import{Text,View}from'react-native';

exportdefaultclassAppextendsReact.Component{

render(){

return(

<Viewstyle={{borderTopWidth:30}}>

<Text>OpenupApp.jstostartworkingonyourapp!</Text>

</View>

);

}

Thiscodeshouldmostlylookfamiliar,asyouhavetheimportstatementsand

get the React usage from that, as well as the View and Text components. The

App uses the Component class and implements a render function that you see

returningtheViewwiththeTextinside.

Figure110–Simplestarterapplication

There are around three dozen components to use to construct your

application.Thesecan befound inthe onlinedocumentationfor ReactNative.

Thissectionwillonlycoverjustafewofthem.Youwillseemoreoftheusage

oftheseintheNewsWatchercode.

ThereisaTextInput component thatis used when you wantto collect text

from a user. It has a prop called onChangeText that you provide a callback

functionto,forcapturingthetextthatisentered.

ThereisaButtoncomponentanditisexactlywhatyouwouldthinkitis.It

hasanonPresspropthatyouprovideacallbackfunctionfor.Thiswillrunonthe

buttonpress.

There are other components for touch interaction, such as

TouchableHighlight, TouchableOpacity and TouchableElement. For example,

youmighthaveanimagethatcanbetouched,orabunchofitemsinalistthat

each have their own touch response. The onPress prop is used to provide a

callbackthatcanbeusedtoprocessthetouchevent.

Thereareseveralcomponentsyoucanusetodisplaylistsofinformation.If

you have a small set of data to render in a list, you can use the ScrollView

component. You will have a performance problem if the list has a lot of

elements.

There is a component named FlatList that only renders the data that is in

view as the user scrolls. You can also use the pagingEnabled prop for paging

throughdataandfetchingitondemandinbatchesasscrollinghappens.

There are many more components to investigate, such as for displaying

images,radiobuttons,checkboxes,selectionpickingandmuchmore.Readthe

official documentation to learn about all that is available

(https://facebook.github.io/react-native/docs/components-and-apis.html).

Note:Itmaysoonbepossibletohaveacommoncomponentlibrarythatcan

evenbeusedforyourWebSiteReactcodeandyourReactNativeapplication.

YouwillmostlikelystillneedtomakesomemodificationstotheUIdesignonthe

mobileapplicationbecauseofthescreensize.

20.3Stylingyourapplication

As was mentioned, HTML elements are not available to use in a React

Native application. CSS is also not available. Instead of using CSS, there is a

specialwayofapplyingstylerightintheJavaScriptyouwrite.Todothestyling,

each core components of React Native all accept a prop named style. This is

closeto how CSS works on theweb, however, names are writtenusing camel

casing.Forexample,youusebackgroundColorinsteadofbackground-color.

While you can use a JavaScript object right inside of the prop, it is much

cleanertosplitthisoutandusetheStyleSheet.createfunctiontodefineseveral

stylesinoneplace.Youthenreferencethatinyourrenderfunctioncode.Hereis

an example that shows the embedded object and the usage of the function

mentioned:

importReactfrom'react';

import{StyleSheet,Text,View}from'react-native';

exportdefaultclassAppextendsReact.Component{

render(){

return(

<View>

<Viewstyle={{width:50,height:50,backgroundColor:'blue'}}/>

<Textstyle={styles.myTextStyle}>Thisissometexttoexperimentwith</Text>

</View>

);

}

conststyles=StyleSheet.create({

myTextStyle:{

color:'blue',

fontWeight:'bold',

fontSize:12,

});

Widthandheight aresetwith propertiesand usevaluesthat willcausethe

sizingtobe thesameacross devices,regardless ofresolution.There isa View

componentusedto drawablue rectangle.Therectangle areais50 by50. The

Text component brings in the style from the returned object of the

StyleSheet.create()call.Therectangleandthetextarebothblue.Thefollowing

imageshowshowthiswilllook.

Figure111-Styling

20.4Layoutwithflexbox

Flexbox is a standard that exists for web page layout. You can read the

Mozilla(MDN) documentation to learn all the fine details. React Native takes

thisstandardandmakesuseofitwithinitsstylingcapability.Whatthisadds,are

propertiesyoucanaddtoyourstylingobjectthatspecifyexactlayoutareas.

Thefirstpropertytoknowaboutistheflexproperty.Thisiswhatyoucan

use to specify relative sizing of anything. For example, you can have three

rectanglesandgivethemeachaflexvalueof1andthenrelativetoeachother,

theywillgetsized.Sincetheyeachhaveavalueofone,theyeachtakeupthe

sameamountofspace.

Thislayout withflexbox can behierarchical, meaning thata parent canbe

part of a flex layout at its level, and then inside of that there, is another flex

layoutthatisapplied.

By default, the flex layout is determining the vertical layout of the area

(called column layout). You can alter that and make it specify the horizontal

layout(calledrowlayout).Youcanalsospecifythejustification,alignmentand

spacing.

Hereissomesamplecodethathastwosubsections.Onehasaflexvalueof

twoandtheotheravalueofone.Thismeansthatthetopareawillbetwicethe

sizeofthelowerarea.Studythefollowingcode:

importReactfrom'react';

import{StyleSheet,Text,View}from'react-native';

exportdefaultclassAppextendsReact.Component{

render(){

return(

<Viewstyle={styles.container}>

<Viewstyle={styles.subContainer1}>

<Textstyle={styles.childText}>One</Text>

<Textstyle={styles.childText}>Two</Text>

<Textstyle={styles.childText}>Three</Text>

</View>

<Viewstyle={styles.subContainer2}>

<Textstyle={styles.childText}>Four</Text>

<Textstyle={styles.childText}>Five</Text>

<Textstyle={styles.childText}>Six</Text>

</View>

);

}

conststyles=StyleSheet.create({

container:{

flex:1,

marginTop:25

subContainer1:{

flex:2,

flexDirection:'row'

subContainer2:{

flex:1

childText:{

flex:1,

borderWidth:2,

textAlign:'center',

fontSize:16

});

YoualsoseethatoneoftheflexboxareashasaflexDirectionsetto‘row’.

Thismakesthecomponentsinsideofitbespreadouthorizontally.Hereiswhat

theresultlookslike:

Figure112-Flexboxlayouthierarchy

Therearemanymoreflexboxstylingpropertiesyoucantakeadvantageof.

Themainpointofflexboxistogetyouawayfromspecifyingpixeldimensions

for anything and having everything sized relative to each other. That way no

matterthedevice,itcanbedisplayedproperly.

20.5Screennavigation

ReactNativedoesnothaveabuilt-incapabilitytosetupanykindofscreen

transitions. Instead, there is a very popular npm library that you can install to

handlethis.Youcandoannpminstallofreact-navigationandthenusethattoset

up different types of navigation. For example, if you want to treat your

navigationlikeastack,whereyoucangofromonescreentoanotherandasyou

transition,thepreviousscreensstayinthestackhistorythatyoucangobackto.

Here is a simple example that shows having a screen with a clock on it and

anotherscreenthatshowstheweatheronit.

import{StackNavigator}from'react-navigation';

constApp=StackNavigator({

Clock:{screen:ClockScreen},

Weather:{screen:WeatherScreen},

});

The ClockScreen and WeatherScreen are just components with a render

function to display whatever you can imagine for each of those screens. The

importantcodetoseeissimplythesettingupoftheuseoftheStackNavigator.

Thisallowsyoutohavecodethatcallsanavigatefunctiontogotoeitherofthe

screensandevenpassinparameters.ThebackbuttononanAndroiddevicewill

alsotakeyoubackascreeninthestack.

The header provided by StackNavigator will contain a back button to go

backfromtheactivescreenifthereisone.

The other mode of navigation is that of using tabs. You see this type of

navigationallthetimeinmobileapplications.Youwillseethetabsatthetopor

thebottom.Thecodeisexactlythesame,butinsteadofusingStackNavigator,

youuseTabNavigator.Once youdoso,you willseethetabappearfor youto

makeyourscreenselection.Youcanaddstylingtothistoaddicons.

The navigation can be set up to be nested. For example, you can have the

main navigation with tabs, but within each screen for the tab, you can set up

somestacknavigation,suchashavingatabwithalist,andclickingonanitem

inthelistdoingastacknavigationtoadetailsscreen.

20.6Devicecapabilityaccess

TheReactNativeAPIisstillgrowinginitscapabilities.Thisistrueasfaras

being able to access the capabilities of the phone hardware. For example, you

canaccessthephone’scamerarollofphotos,butnotthecameratotakeaphoto.

Youcangetatthegeolocation,butnotattheaccelerometer.Itcanbeassumed,

thatovertime,mostphonecapabilitieswillbeexposedthroughtheReactNative

API.

Until that time, you can simply use the API provided by Expo. Besides

providingtheUItocreateandrunapplications,therehasbeenprovidedalibrary

in npm that you can use in your code to access thing like the camera,

accelerometer,fingerprint,gyroscope,contactsandmuchmore.

TousetheCameracomponent,youuseitlikeotherReactcomponents.This

meansthatyouwillneedtorenderitasanelementfirstinsomeJSX.Toactually

take a photo, you need to get a reference to the component and then call a

functiononit.Hereiswhatitwouldlooklike.

//intherender()function

<Cameraref={ref=>{this.camera=ref;}}/>

//Somewhereincode,perhapsinresponsetoatouchevent

this.camera.takePictureAsync().then(data=>console.log(“Tookaphoto”));

20.7CodechangestoNewsWatcher

You can now learn about the code changes that are needed to get the

NewsWatcherapplicationwrittenasanativeapplication.

Thestartingpoint

AswithourotherReactcode,thiscodealsohasanApp.jsfilethatactsasthe

startingpointforyourlogictorenderandrunallyourcode.

Thecodewillsetupthenavigation,includingtheUIforthenavigationbar

andalltheindividualscreencomponentsthatwillmakeuptheapplication.The

navigationbar is dependanton the mobile platform,because iOS andAndroid

UI navigation can look different.Thus, there is code that determines which to

render.ThePlatform.ospropertyholdsastringthatwilltellyouwhichmobile

platformyouarerunningon.

YouwillseeReduxbeingusedintheeactsamewayasbefore.Hereisallof

thecodeforthestartup.

//App.js

importReactfrom'react';

import{Platform,StatusBar,StyleSheet,View,AsyncStorage}from'react-native';

import{AppLoading,Asset,Font}from'expo';

import{Ionicons}from'@expo/vector-icons';

import{TabNavigator}from'react-navigation';

importMainTabNavigatorfrom'./navigation/MainTabNavigator';

import{createStore}from'redux'

import{Provider}from'react-redux'

importreducerfrom'./reducers'

import{fetchMyNews}from'./utils/utils';

import{fetchMyProfile}from'./utils/utils';

conststore=createStore(reducer);

constRootNavigator=TabNavigator(

{

Main:{

screen:MainTabNavigator,

{

navigationOptions:()=>({

headerTitleStyle:{

fontWeight:'normal',

}),

}

);

exportdefaultclassAppextendsReact.Component{

state={isLoadingComplete:false};

componentDidMount(){

//Checkfortokenindevicelocalstorage,meaninguserissignedin

AsyncStorage.getItem('userToken',function(err,value){

if(value){

consttokenObject=JSON.parse(value);

store.dispatch({ type: 'RECEIVE_TOKEN_SUCCESS', msg: `Signed in as

${tokenObject.displayName}`,session:tokenObject});

fetchMyNews(store.dispatch,tokenObject.userId,tokenObject.token);

fetchMyProfile(store.dispatch,tokenObject.userId,tokenObject.token);

}

})

}

render(){

if(!this.state.isLoadingComplete&&!this.props.skipLoadingScreen){

return(

<AppLoading

startAsync={this._loadResourcesAsync}

onError={this._handleLoadingError}

onFinish={this._handleFinishLoading}

);

}else{

return(

<Providerstore={store}>

<Viewstyle={styles.container}>

{Platform.OS==='ios'&&<StatusBarbarStyle="default"/>}

{Platform.OS==='android'&&<Viewstyle={styles.statusBarUnderlay}/>}

</View>

</Provider>

);

}

_loadResourcesAsync=async()=>{

returnPromise.all([

Asset.loadAsync([

require('./assets/images/robot-dev.png'),

require('./assets/images/robot-prod.png'),

]),

Font.loadAsync({

//Thisisthefontthatweareusingforourtabbar

...Ionicons.font,

'space-mono':require('./assets/fonts/SpaceMono-Regular.ttf'),

}),

]);

};

_handleLoadingError=error=>{

//Inthiscase,youmightwanttoreporttheerrortoyourerror

//reportingservice,forexampleSentry

console.warn(error);

};

_handleFinishLoading=()=>{

this.setState({isLoadingComplete:true});

};

}

conststyles=StyleSheet.create({

container:{

flex:1,

backgroundColor:'#fff',

statusBarUnderlay:{

height:24,

backgroundColor:'rgba(0,0,0,0.2)',

});

NavigationUI

The react-navigation module is used that requires an object as part of its

construction.Therearedifferenttypesofnavigationsavailable,suchasstackand

tab. We will be using tab by using the TabNavigator function. You pass in an

objecttothatandspecifythefilesthatholdthescreenUIsandalsothetabsand

iconsforthetoolbartouseforeach.

HereisthecodefortheconfiguringofthenavigationfortheNewsWatcher

application:

//MainTabNavigator.js

importReactfrom'react';

import{Platform}from'react-native';

import{Ionicons}from'@expo/vector-icons';

import{TabNavigator,TabBarBottom}from'react-navigation';

importColorsfrom'../constants/Colors';

importHomeScreenfrom'../screens/HomeScreen';

importMyNewsScreenfrom'../screens/MyNewsScreen';

importLoginScreenfrom'../screens/LoginScreen';

importProfileScreenfrom'../screens/ProfileScreen';

exportdefaultTabNavigator(

{

HomeNews:{

screen:HomeScreen,

MyNews:{

screen:MyNewsScreen,

NewsFilters:{

screen:ProfileScreen,

Account:{

screen:LoginScreen,

{

navigationOptions:({navigation})=>({

tabBarIcon:({focused})=>{

const{routeName}=navigation.state;

leticonName;

switch(routeName){

case'HomeNews':

iconName=

Platform.OS==='ios'?

`ios-home${focused?'':'-outline'}`

:'md-home';

break;

case'MyNews':

iconName=

Platform.OS==='ios'?

`ios-funnel${focused?'':'-outline'}`

:'md-funnel';

break;

case'NewsFilters':

iconName=

Platform.OS==='ios'?

`ios-options${focused?'':'-outline'}`

:'md-options';

break;

case'Account':

iconName=

Platform.OS==='ios'?

`ios-person${focused?'':'-outline'}`

:'md-person';

}

return(

<Ionicons

name={iconName}

size={28}

style={{marginBottom:-3}}

color={focused?Colors.tabIconSelected:Colors.tabIconDefault}

);

}),

tabBarComponent:TabBarBottom,

tabBarPosition:'bottom',

animationEnabled:false,

swipeEnabled:false,

}

);

TheiconNamepropertycontainsavaluespecifyingthevectorgraphicicon

touse.TheseneedtobesetforvaluesdependingoniOSandAndroid.Thereisa

website to go to for seeing the list of possible icons. You can search the site

https://expo.github.io/vector-icons/.

NowyoucanconsidereachofthescreensthataresimplyReactComponent

derivedclassesthatReactNativenavigationusestorenderastheyarebrought

intousage.

TheHomeNewsscreen

Thisiscodeforshowingthehomepagescreenthatcontainsthedefaulttop

news stories. You will see much that is familiar. You see that there is still a

componentDidMount()function that contains the backend fetch call.The local

componentstateisusedforthatstorage.

In the scrollable list of news stories, you see something new. This is the

TouchableNativeFeedback React Native component. This is used as the outer

parentforallofwhatiscontainedinsideandgivestheabilitytocaptureatouch

thatwilllaunchthebrowsertodisplaythenewsstory.Everythingelseisafairly

straightforward translation from the HTML elements to the ones that are

supportedbyReactNative.

//HomeScreen.js

importReactfrom'react';

import{

Image,Platform,StyleSheet,

Text,TouchableHighlight,

TouchableNativeFeedback,View,

ScrollView,Alert

}from'react-native';

import{WebBrowser}from'expo';

import{toHours}from'../utils/utils';

exportdefaultclassHomeScreenextendsReact.Component{

constructor(props){

super(props);

this.state={isLoading:true,news:null};

}

staticnavigationOptions={

title:'HomeNews',

};

componentDidMount(){

returnfetch('https://www.newswatcher2rweb.com/api/homenews',{

method:'GET',

cache:'default'

})

.then(r=>r.json().then(json=>({ok:r.ok,status:r.status,json})))

.then(response=>{

if(!response.ok||response.status!==200){

thrownewError(response.json.message);

}

for(vari=0;i<response.json.length;i++){

response.json[i].hours=toHours(response.json[i].date);

}

this.setState({

isLoading:false,

news:response.json

});

})

.catch(error=>{

this.props.dispatch({type:'MSG_DISPLAY',

msg:`HomeNewsfetchfailed:${error.message}`});

Alert.alert(`HomeNewsfetchfailed:${error.message}`);

});

}

onStoryPress=(story)=>{

WebBrowser.openBrowserAsync(story.link);

};

onNYTPress=()=>{

WebBrowser.openBrowserAsync('https://developer.nytimes.com');

};

render(){

if(this.state.isLoading){

return(

<View>

<Text>Loadinghomepagenews...</Text>

</View>

);

}

letTouchableElement=TouchableHighlight;

if(Platform.OS==='android'){

TouchableElement=TouchableNativeFeedback;

}

return(

<View>

{this.state.news.map((newsStory,idx)=>

<TouchableElementkey={idx}

onPress={()=>this.onStoryPress(newsStory)}>

<Viewstyle={styles.row}>

<Viewstyle={styles.imageContainer}>

<Imagesource={{uri:newsStory.imageUrl}}

style={styles.storyImage}/>

</View>

<Viewstyle={styles.textContainer}>

<Textstyle={styles.storyTitle}numberOfLines={2}>

{newsStory.title}

</Text>

<Textstyle={styles.storySnippet}numberOfLines={3}>

{newsStory.contentSnippet}

</Text>

<Textstyle={styles.storySourceHours}>

{newsStory.source}-{newsStory.hours}

</Text>

</View>

</TouchableElement>

)}

<TouchableElementkey={this.state.news.length}

onPress={()=>this.onNYTPress()}>

<Viewstyle={styles.row}>

<Image

source={require('../assets/images/poweredby_nytimes_30b.png')}

<Viewstyle={styles.textContainer}>

<Textstyle={styles.storyTitle}numberOfLines={2}>

DataprovidedbyTheNewYorkTimes

</Text>

</View>

</TouchableElement>

</ScrollView>

</View>

)

}

conststyles=StyleSheet.create({

row:{

alignItems:'center',

backgroundColor:'white',

flexDirection:'row',

borderStyle:'solid',

borderBottomColor:'#dddddd',

borderBottomWidth:StyleSheet.hairlineWidth,

padding:5,

imageContainer:{

backgroundColor:'#dddddd',

width:90,

height:90,

marginRight:10

textContainer:{

flex:1,

storyImage:{

width:90,

height:90,

storyTitle:{

flex:1,

fontSize:16,

fontWeight:'500',

storySnippet:{

fontSize:12,

marginTop:5,

marginBottom:5,

storySourceHours:{

fontSize:12,

color:'gray',

});

HereiswhatthatUIlookslikeonmyphonefothehomenewsscreen:

Figure113–Homenewsscreen

IfyouarelookingcloselyandcomparingthecodeoftheReactWebappwith

that of the React Native app, you will notice that we don’t actually use the

componentDidMount() function in the React Native code in some places. You

willseethattheProfileScreenandMyNewsScreencomponentsdon’thavethat

functionliketheHomeScreencomponentdoes.

Thereareafewreasonswhy.Onedifferenceisthatthisgetscalledonlyonce

and it gets called even before you switch to each screen. For example, it gets

calledontheprofilescreenatthestartuptimeoftheappeventhoughtheprofile

screenisnotbeingshown.Thefetchneedstomakeuseofthesessiontokenat

thattimetoget dataforthe user.The problemisthat thereisarace condition

with the App.js that gets the token from the local storage and it might not be

available yet. The fix is to place the fetching when you know the session is

available.

TheHomeScreencomponentdoesnotneedaJWTtokentocalltheendpoint

APIitisusing,sothereisnotimingissue.

Theotherscreens

Similarcodetransformationshappenformakingtheotherscreens.Youcan

look at the code for the LoginScreen, MyNewsScreen and ProfileScreen

components.Iwillgiveafewhighlightsandgiveyouscreenimagestolookat.

Modal:YouwillseetheusageoftheModalcomponent.Inthiscase,itisa

bitdifferentfromwhatwedidintheReactwebcode.Here,thereisactuallya

visiblepropthatwesettocontrolifitisvisibleornot.

Buttons:YouwillseeinthecodethatIexperimentedwithmakingbuttons

using both the Button and the TouchableOpacity components. The latter is

simplymoreversatile,butbothbehaveandlooksimilar.

HereiswhatthatUIlookslikeonmyphoneforamodalcomponentandthe

buttons:

Figure114–ModalUI

Keyboard:TheKeyboardAvoidingViewcomponentisusedtowrapUIthat

willmoveoutofthewaywhenakeyboardinteractionUIismovedintoplace.

This is used around the TextInput components. The TextInput themselves also

have some interesting props to use to do things like stating what type of

keyboardtodisplayandhowtotransitiontoanotherfield.

HereiswhatthatUIlookslikeonmyphoneforthekeyboardusage:

Figure115–KeyboardUI

Picker:ThePicker component is used to display a type of drop-down for

makingselections.Theselections,inthiscase,arethenewsfiltersthatyoucan

setupandchoosebetween.

HereiswhatthatUIlookslikefortheprofilescreenwheretheuserspecifies

theirnewsfilters:

Figure116–NewFilterprofileUI

Applicationconfigurationfile

ThereisaspecialfilethatisprovidedthatcontrolssomeaspectsoftheReact

Nativeapplication. Youcan openand look at the app.jsonfile to view whatit

contains.Youcanseethat youspecifythingslikethesplash screenimageand

alsowhatthedescriptionisfortheapplicationasshownintheappstore.Oneof

the things I needed to do for the Android section was to specify the

WAKE_LOCKpermission.Thisissettokeepthescreenfromdimmingandthe

processorfromgoingtosleep.

//app.json

{

"expo":{

"name":"NewsWatcher",

"description":"Newsservingapplication",

"slug":"NewsWatcher",

"privacy":"public",

"sdkVersion":"23.0.0",

"version":"1.0.0",

"orientation":"portrait",

"primaryColor":"#cccccc",

"icon":"./assets/images/icon.png",

"splash":{

"image":"./assets/images/splash.png",

"resizeMode":"contain",

"backgroundColor":"#ffffff"

"ios":{

"supportsTablet":true

"android":{

"versionCode":2,

"package":"com.blueskyproductions.newswatcher2rweb",

"permissions":["WAKE_LOCK"]

}

20.8ApplicationStoreDeployment

Aftertheapplicationisalldeveloped,youwillwanttomakeitavailablein

oneoftheapplicationstores.Iwillwalkthroughsomeofthedetailsyouneedto

knowtogetthisupandontheGooglePlaystore.Whatyouwilldoiscreatethe

APKstandalonebinarythatisusedtodistributeyourapplication.

When using Expo, the standalone app knows to look for updates at your

app's published URL. You can later run a command (exp publish) to upload a

new version. The next time a user opens the app they will automatically

downloadthenewversion.Thisiscalledan"OvertheAir"(OTA)updateandis

builtintoExposoyoudon'tneedtoinstallanything.

To build the APK (or iOS IPA file), open a Node.js command prompt. A

regular windows DOS prompt is not capable of running the commands. You

needtorunthefollowingcommandinthedirectoryofyourproject.

expbuild:android

This process will transfer the application code up to a cloud server run by

Expoandabuildprocesswillrunthere.Youcanseethatitisprogressingifyou

go to the link provided for you each time. Once a build is done, you can also

install it directly from the Expo portal with the Expo app on your phone by

snappinganimageoftheQRcodeyoucanview.

Figure117-Buildoutput

It could take five or so minutes to complete. Once it is done, you are

presentedwithaURLtousetodownloadtheAPKfileinthecaseofanAndroid

build.YourunasimilarcommandtocreateafileforiOS.

Onceyouhavedownloadedthefile,youcanuploadittotheappstore.Toget

thatupandavailabletoAndroidusersthroughtheGooglePlaystoreyouneedto

haveanaccounttodosoandgototheportaltoworkthroughthat.Onceyour

application is available in the store, you can see it at

https://play.google.com/store and of course open thePlay Store on your phone

andinstallit.

To upload your application file for Android and publish it on Google Play

youcangotohttps://play.google.com/apps/publish.Youcanclickthebuttonto

create a new application. You can begin by publishing your app as an Alpha

release.YoucanclickonReleasemanagementonthe leftmenuare andthen

chooseAppreleases.Thenonthatscreen,youwillseewhereyoucanclickto

managetheAlpharelease.Youcancreateareleaseanduploadyour.APKfile.

You can continue to upload newer updates to your APK file as you add

features and fix bugs. Alternatively, Expo can be used to publish releases that

don’trequireyoutogothroughtheGooglePlayportal,butareupdatesthatthe

applicationwillautomaticallydownloadonthemobiledevice.

OnceyouhavetheAPKfileuploadedtoGooglePlay,youcanclickonthe

Alpharelease and add testers to it. Youcan email a specificURLthat will be

availabletoanyfriendsthatwouldbeabletohelptesttheapplication.Youcan

then promote it to Beta and finally to Production where it will be seen by

everyoneintheGooglePlayapplicationstore.

If you have an Android phone, you can open Google Play and search for

“NewsWatcher”andinstalltheapplicationIuploadedtherealready.

Note:TherearestillafewkinkstoworkouttogetEnzyme,withJesttobe

abletodoamountandshallowrenderingintestcodewiththem.Thatwouldbe

agreatresearchprojectonitsown.ThereisdocumentationinEnzymeforthis,

butitstilldoesnotcarryitdeepenoughtobeabletodotestingofReactNative

asitdoeswithReactfortheweb.

LettingGooglePlaymanagethepublicsigningkey

ThereisonethingIomittedtomakeclearintheprevioustext.Thatisthatif

you upload the application with the Expo signed key, you can't ever upload

anotherAPKfile,asGooglePlaywillcomplainaboutanumberofthings.For

example,it will complain that the same versionnumber is being used already.

Butifyouchangethat,thenExpogenerates anewkeyforitand GooglePlay

willnotacceptit.ReadtheExpodocumentationonthisandtheymakeitclear

thatyoucanoptintoGooglePlaydoingtheusersignedkeyandthenyoucanset

upExpotoletyoumanagesigningwithaprovideduploadkeythatyougiveit.

Bottomline,itseemslikeyouwouldwanttogowithGooglePlaymanaging

thesigningsothatyoucanuploadnewAPKfiles.Otherwise,youaretiedinto

using Expo always to do the updates in the app through their mechanisms. I

knowthisallsoundsconfusingatfirst,sojusttryandfollowalonghere.

To start with, you need to open Android Studio and just create a new

application. Then once that is created, click on the Build menu and select

GenerateSignedAPK.TherewillbeabuttontherenamedCreateNew.Click

thatandfillintheform,suchasfollows:

Figure118–Creatinganuploadsigningkey

Follow through and do the APK generation. You will not be needing the

APKfile, butwill be takingthe jks fileand using thatwhen you dotheExpo

build.Makesuretotakenoteofthepasswordsandthealiasyoutypedin.Now,

whenyouruntheExpobuildcommandandselecttoprovideyourownkey,you

willbeaskedforthepathandnameofthejksfile.Followthepromptsandyou

willgetanAPKfileyoucanusewithGooglePlaynow.Youwillusethisfile

overandovertodothesigningeverytimeyouwanttouploadanewversionof

yourapplicationtoGooglePlay.

Note:IutilizeExpointhisbookbecauseitistheeasiestandfastestwayto

getsomethingupandworking.Youmaydecidethatyouwantthefullflexibility

tocreateandmanageyourReactNativeapplicationyourself.Inthatcase,you

shouldcreate the applicationwith create-react-native-app installedwith NPM.

Youinstallthisandcandoeverythingyouneedfromthecommandline,oreven

withtheuseofAndroidStudio.

CONCLUSION

A lot of material has now been covered and you should at this time feel

really good about your newly acquired knowledge and skills. You started by

learningaboutwhatathree-tierarchitectureisandhowaMERNJavaScriptfull-

stackimplementationisagreatchoiceforanimplementation.

TheNewsWatchersampleapplicationwasfeaturedthroughoutthebookasa

fullend-to-endimplementationofthearchitectureyouhavebeenlearningabout.

Younowhavethecodethatyoucanrefertoandbaseanyofyourownworkon.

Thethree partsof the book coveredthe layers ofa three-layerarchitecture

andgaveyoufoundationalknowledgeabouteachlayer.Youalsolearnedabout

theSDKstouseandspecificusagescenariosofeverythingfortheNewsWatcher

application.

You can now go forward and do your own learning for your own specific

needsandbesuccessful.Youmaychoosetojustworkinone ofthelayers,or

you may be able to contribute across all layers. It is always good to be

knowledgeableoverallasyouwillknowbetterhoweverythingworksinternally

andrealizehowwhatyoudoinonelayeraffectsanotherlayer.